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FOREWORD 


The  research  reported  here  is  one  experiment  conducted 
under  one  task  of  Project  4690,  Threat  Evaluation  and  Action  Selection. 
Until  the  Fall  of  1961,  the  technical  direction  of  this  project  was  the 
responsibility  of  the  Detection  Physics  Laboratory  of  AFCRL.  This 
project  was  conceived  and  prosecuted  by  the  above  Laboratory  under 
the  direction  of  Mr.  W.  H.  Vance,  Jr.  The  particular  task  under 
which  this  research  was  accomplished,  Task  46902,  Data  Processing 
for  Threat  Evaluation,  was  the  technical  responsibility  of  the 
Components  and  Techniques  Division  of  the  Operational  Applications 
Laboratory  of  ESD.  The  present  research  was  the  second  major 
experiment  in  a  series  designed  to  study  the  performance  capabilities 
and  limitations  of  skilled  and  operationally  experienced  tactical 
decision  makers  in  a  simulated  aerospace  surveillance  system.  This 
document  is  the  full  detailed  report  of  the  experiment,  part  of  which 
has  been  previously  described  in  an  executive  summary  report.  It  is 
hoped  that  the  results  will  yield  information  which  will  assist  in  the 
.  design  of  future  aerospace  surveillance  systems  and  the  optimum 
utilization  of  human  talents  within  such  systems. 
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ABSTRACT 


An  experiment  was  performed  in  which  five  highly  exper¬ 
ienced  subjects  were  required  to  perform  threat  evaluation  and  action 
selection  functions  under  aerospace  surveillance  loads  of  from  60  to 
96  incoming  tracks.  Other  influential  conditions  were  the  overall 
flight  performance  level  of  the  threats  and  the  quality  of  the  surveill¬ 
ance  data  presented  to  the  experimental  commanders. 

The  most  generally  influential  condition  was  task  load, 
increases  in  which  caused  increased  weapon  consumption,  an  increas¬ 
ing  but  negatively  accelerated  rate  of  kill  of  threats,  increasing  and 
positively  accelerating  amounts  of  damage  and  increased  reaction 
time.  The  load  build-up  rate  beyond  which  commanders  began  to  lag 
behind  in  the  selection  of  counter  actions  was  found  to  be  of  the  order 
of  5-6  tracks  per  minute.  The  performance  level  of  the  incoming 
threat  did  not  produce  clear-cut  evidence  of  effects  upon  the  comman¬ 
der's  success  at  their  tasks.  Tracks  whose  position  and  identifying/ 
descriptive  data  were  50%  -  60%  complete  and  correct  were  handled 
in  about  the  same  way  as  tracks  represented  by  perfect  information. 

The  commanders  made  only  small  numbers  of  inappropriate 
action  selections.  While  the  load  range  tested  here  began  to  cause 
deterioration  of  action  selection  performance,  no  drastic  break  point 
was  found  for  any  measure.  It  was  found,  however,  that  commanders 
based  their  actions  upon  only  the  broadest  criteria,  (e.g.  ,  threat 
vehicle  class)  and  did  not  (or  were  unable  to)  make  fine  discrimina¬ 
tions  of  relative  or  absolute  threat. 
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INTRODUCTION 


The  experiment  reported  here  is  the  second  major  experiment 
in  a  series  devoted  to  detailed  study  of  man-machine  performance  in 
the  aerospace  command  and  control  tasks  of  threat  evaluation  and 
action  selection  (TEAS).  Aerospace  surveillance  could  be  divided, 
logically,  into  several  major  areas  or  phases.  The  first  of  these  may 
be  thought  of  as  raw  data  acquisition,  possibly  including  filtering  and 
first  order  processing.  The  transmission  of  data  from  collectors  to 
more  central  processors  can  be  thought  of  as  distinguishable  frcm 
collection  and  processing  or  as  part  of  either  or  both.  A  further  area 
or  phase  in  the  surveillance  process  might  be  considered  to  involve 
data  display,  evaluation  and  utilization.  A  final  phase  may  be  thought 
of  as  the  implementation  of  command  decisions  or  the  "control"  pro¬ 
cesses.  The  series  of  experiments  of  which  this  is  a  part  has  been 
aimed  at  study  of  the  evaluation  and  utilization  processes,  specifically 
aerospace  surveillance  tactical  decision  making. 

In  order  to  study  this  one  phase  of  the  over -all  mission  of 
aerospace  command  and  control  in  comparative  isolation  and,  as  far 
as  possible,  under  controlled  and  controllable  conditions  it  has  been 
necessary  to  simulate  the  whole,  integrated  process.  Early  studies 
(2)  were  devoted  to  testing  the  feasibility  of  certain  approaches. 

Other  studies  (3,  5)  have  been  directed  to  the  modification  and  im¬ 
provement  of  the  simulation  facility. 

The  over-all  background  of  the  whole  program  requires  quite 
voluminous  treatment,  which  is  given  in  prior  reports  (2,  4)  and 


which  will  not  be  repeated  here.  It  is  sufficient  to  state  at  this  point 
that  the  objectives  of  this  research  have  been  to  assay  the  possibilities 
of  "providing  fundamental  knowledge,  concepts,  methods  and  techni¬ 
ques  on  which  to  base  development  of  (surveillance)  equipment  and 
systems  at  some  future  time."  (4)  More  specifically,  it  has  been  the 
objective  of  these  experiments  to  investigate  man-machine  capabilities 
in  surveillance  decision  making  and  even  more  particularly,  human 
performance  of  complex  command  and  control  types  of  activities.  The 
value  of  knowledge  of  human  capabilities  and  limitations  in  this  type 
of  behavior,  if  it  can  be  obtained,  and  of  the  general  and  specific  func¬ 
tions  whereby  such  tasks  are  accomplished  should  be  two -fold.  At  the 
very  least  it  should  cast  light  on  reasonable  (if  not  optimum)  means  of 
aiding  such  human  capabilities  or  of  using  them  to  best  advantage  in 
systems  of  the  future.  In  addition,  even  general  fundamental  insights 
into  human  performance  in  this  area  are  desirable  if  not  essential  to 
any  future  "automation"  of  all  or  part  of  the  decision  making  task. 

The  first  major  experiment  in  this  series  was  concerned  with 
the  study  of  the  effects  of  over -all  task  load  upon  the  performance  of 
"tactical  decision  makers"  under  circumstances  where  the  information 
supplied  them  was  essentially  complete  and  correct.  In  addition,  the 
threats  which  they  were  required  to  evaluate  and  counteract  were  of  a 
conventional,  air-breathing  sort.  The  major  task  of  the  "commanders" 
was  the  selection,  from  a  varied  inventory  of  varying  size,  of  an 
appropriate  choice  of  counter  weapons,  where  the  outcome  of  any  action 
was  more  or  less  uncertain.  Actual  evaluation  of  threat  in  relative  or 
absolute  terms  was  only  a  small  part  of  their  jobs.  The  latter 


-2- 


,a  ■  *  -  V-  »  '  l'  — V*-,-*!  i  ,  r  “-.  '»  -1  ■  , 


experiment  is  described  in  a  detailed  technical  report  (4). 

The  present  experiment,  the  second  major  one  in  the  series, 
was  intended  to  study  the  effects  on  the  man-machine  decision  makers 
of  two  additional  variables  besides  sheer  task  load.  One  variable 
consisted  of  two  levels  of  threat  --  slower,  lower  altitude,  somewhat 
less  "lethal"  vehicles  on  the  one  hand  and  faster,  higher  altitude, 
more  lethal  vehicles  on  the  other.  The  detailed  characteristics  of  the 
threats  and  the  counter  weapons  may  be  found  in  Appendix  I.  The  other 
variables  consisted  of  awide variety  of  kinds  and  amounts  of  informa¬ 
tion  reliability.  More  details  of  the  design  of  the  present  experiment 
may  be  found  in  Appendix  II. 

Both  of  these  additional  variables  were  intended  to  sample  the 
effects,  on  tactical  decision  making,  of  threat  evaluation  and  action 
selection  situations  which  were  more  "futuristic"  (in  terms  of  weapon 
performance)  and  more  "realistic"  (in  terms  of  surveillance  system 
performance)  than  had  been  the  case  in  previous  experiments.  The 
first  major  experiment  was  intended,  in  part,  to  establish  some  "base¬ 
lines"  of  decision  making  performance.  The  second  was  intended  to 
ascertain  the  above  effects,  if  any,  and  to  obtain  some  insight  into  the 
widely  known  abilities  of  the  human  to  make  surprisingly  good  use  of 
unreliable  and  incomplete  information.  Both  of  these  and  succeeding 
experiments  were  designed  to  study  a  complex,  realistic  and  militarily 
relevant  decision  making  situation  as  opposed  to  the  simpler,  "neater" 
and  more  tightly  controllable  situations  which  have  been  the  forte  of 
more  basic,  academic  and  theoretical  endeavors.  Both  approaches 
are  necessary;  neither  is  perfect.  The  "design"  employed  here  is  not 
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of  a  complete  classical  sort.  While  a  structure  of  independent  and 
dependent  variables  existed  and  controls  were  exercised  (cf  Appendix 
II  Section  2)  the  experiment  was  not  set  up  as  a  statistical  design. 
There  were  two  major  reasons  for  this.  Firstly,  to  run  a  complete 
or  partially  complete  orthogonal  design  would  have  been  utterly  pro¬ 
hibitive  in  terms  of  time  and  financial  costs.  Secondly,  the  interest 
was  in  the  practical  or  indicative  types  of  results  rather  than  the 
inferential  statistical  type.  It  may  be  noted  that  there  is  no  necessary 
connection  between  statistical  and  practical  11  significance". 

Subsequent  experiments  in  this  series  will  study  the  perform¬ 
ance  of  man  as  a  ‘'monitor"  or  "over-rider"  of  decision  making  of  the 
same  types  as  studied  here  since  it  seems  likely  that  such  a  role  will 
become  increasingly  important  in  command  and  control  systems  of 
the  future. 
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PROCEDURES 


The  basic  apparatus  and  the  adaptations  and  improvements 
which  preceded  this  study  have  been  described  in  great  detail  in 
previous  reports  (5,  2,  4).  For  the  understanding  of  this  study  cnly 
limited  detail  is  necessary.  Schematic  site  layout  and  information 
flow  diagrams  may  be  found  in  Figures  1  and  2.  From  the  position 
of  the  tactical  decision  maker  the  environment  consisted  of  a 
"command  post"  type  of  facility.  The  decision  maker  was  seated 
before  a  console  on  which  were  displayed  to  him  the  geographic  posi¬ 
tion  situation  of  friendly  and  enemy  air  and  space -borne  forces. 
"Targets"  or  tracks  consisted  of  bright  spots  on  a  cathode  ray  tube 
screen  with  a  limited  amount  of  track  history  shown  by  persisting  pre¬ 
vious  positions.  The  tracks  were  capable  of  being  coded  into  groups 
according  to  broad  categories  by  switch  selection  (e.  g.  ,  classified  by 
identification,  height,  weapon  type,  etc.  ).  The  coding  was  effected  by 
dimming,  brightening,  focussing  or  defocussing  track  spots.  Tracks 
appearing  for  the  first  time  or  tracks  whose  identification  or  other 
ancillary  data  had  been  changed  appeared  as  blinking  spots.  Any  track 
could  be  selected  by  number  or  any  track  spot  could  be  interrogated 
with  a  photo-electric  device.  Tracks  so  selected  would  blink,  all 
other  blinking  tracks  would  cease  doing  so  and  all  ancillary  informa¬ 
tion  associated  with  the  particular  track  (IFF,  speed,  altitude,  vehicle 
type,  the  "raid"  of  which  it  was  a  part  if  any,  and  the  raid  size  if  any) 
would  be  read  out  in  alphanumeric  form  in  a  bank  of  small  windows  at 
the  top  of  the  console.  This  track  position  situation  and  the  identifying 
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Figure  2.  Information  Flow  Diagram 


and  descriptive  data  were  supplied  from  digital  recordings  on  magnetic 
tape  which  were  previously  generated  with  track  generating  equipment. 
Additional  detail  on  the  contents  of  the  recorded  threat  situations  may 
be  found  in  Appendices  I  and  II. 

In  addition  to  the  electronically  generated  data  described 
above,  the  decision  maker  had  other  displays.  His  CRT  display  was 
overlaid  with  a  skeleton  diagram  showing  the  location  of  his  defensive 
weapon  sites.  The  total  surveillance  area  measured  300  miles  square. 
There  was  also  a  weapon  status  board  (a  schematic  drawing  of  which 
will  be  found  in  Appendix  III)  and  an  action  selection  log  which  was  kept 
by  the  decision  makers  technician.  The  status  board  in  this  experi¬ 
ment  was  manually  posted  by  two  airmen  technicians  behind  the  edge- 
lit  lucite  board. 

The  rest  of  the  equipment  consisted  mainly  of  interior 
communications  (a  modified  AN/GTA-6)  pencils  and  paper. 

It  might  be  best  to  describe  the  over-all  operating  procedure 
by  describing  portions  of  an  operation  step  by  step  in  time  order. 

1.  The  tape  recorded  track  situation  data  were  fed  into 
the  digital  communication  system,  thence  to  a  drum  store  and  thence  to 
the  situation  display.  The  first  targets  appeared  at  problem  time  zero, 
at  which  time  the  weapon  status  board  appeared  as  shown  in  Appendix 
III. 

2.  The  tactical  commander  would  interrogate  the  first 
tracks  one  at  a  time.  On  the  basis  of  position,  identification,  kind, 
speed,  etc.  ,  he  would  act  against  each  target. 
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3.  The  action  selected  consisted  of  the  track  number 


to  be  attacked,  the  weapon  kind,  the  site  from  which  it  was  to  be 
drawn,  the  armament  type  and  the  number  of  such  weapons.  The  mode 
of  employment  was  assumed  to  be  in  a  single  flight  group  unless  other¬ 
wise  specified  (cf.  also  Appendix  I). 

4.  The  action  selected  was  simultaneously  recorded, 
together  with  the  time  at  which  it  was  selected  by  the  commanders 
airman  technician  and  by  the  ’’scramble  clerk"  at  the  referee  station. 

5.  The  scramble  clerk  passed  the  slip  with  the  action 
data  to  one  of  two  inventory  clerks  (one  for  missiles,  the  other  for 
aircraft  assignments).  Appropriate  deductions  from  inventory  were 
communicated  to  the  weapon  status  board  where  the  board  keepers 
made  the  necessary  changes. 

6.  The  inventory  clerks  passed  the  assignments  to  one 
of  two  referees  (one  for  higher  numbered  tracks,  the  other  for  lower 
numbers)  who  evaluated  the  action  selection.  Referees  had  two  sheets, 
containing  distributions  of  "kills"  and  "misses"  for  nearly  any  action 
which  could  be  selected  for  each  track.  A  sample  referee  sheet  with 
detailed  explanation  may  be  found  in  Appendix  III.  Further  details  of 
general  scoring  procedures  may  be  found  in  Appendix  I. 

7.  The  outcome,  if  a  kill,  was  passed  to  the  closeout 
technician  who  would: 

(a)  cause  the  track  in  question  to  be  removed  from 
the  system  at  the  time  specified  by  the  referee  (this  time  also  was  pre¬ 
planned  and  was  a  function  of  the  intercept  time  which  in  turn  depended 
upon  the  speed  of  the  weapon  assigned  and  the  distance  to  go  at  scram¬ 
ble  time)  and. 
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(b)  transmit  the  outcome  information  to  the  comman¬ 
der  (via  the  commanders  technician)  at  outcome  time. 

8.  The  outcome,  if  not  a  kill,  was  also  transmitted  to 
the  commander  at  the  outcome  time,  but  without  closing  the  track  out 
of  the  system. 

9.  The  damage  clerk  had  a  running  record,  by  time, 
of  all  pre-programmed  potential  damage  which  could  be  inflicted  by 
any  track.  All  assignments,  therefore,  were  passed  through  the 
damage  clerk  position.  If  a  track  capable  of  inflicting  damage  was 
killed  prior  to  causing  the  damage,  no  losses  were  assessed.  Other¬ 
wise,  the  penetrated  defensive  weapon  site  was  closed  due  to  damage 
for  pre-planned  periods  of  time  and/or  defensive  weapons  were  re¬ 
moved  permanently  from  the  inventory  of  the  site.  These  damages 
were  posted  on  the  weapon  status  board. 

Experimental  problem  runs  had  durations  of  35  to  45  minutes, 
depending  on  the  particular  track  situation  presented  and  the  success 
of  the  particular  commander  in  dealing  with  it.  The  details  of  the 
experimental  plan  may  be  found  in  Appendix  II. 

The  commanders  in  the  present  experiment  were  selected 
as  the  five  most  proficient  of  the  nine  who  had  served  in  the  first 
major  experiment.  One  of  the  main  interests  of  this  part  of  the  prort4 
gram  has  been  to  study  as  uniformly  high  a  grade  of  tactical  decision 
making  as  possible,  since  it  has  been  felt  that  good  performance  would 
be  more  revealing  of  genuine  and  general  problems  and  suggestions 
for  their  solution.  The  five  commanders  (or  tactical  decision  makers) 
in  this  experiment  were  all  First  Lieutenants  and  Captains 
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with  extensile  AC&W  experience  in  operational  sites  and  in  the 
Experimental  Sage  Sector  as  well  as  in  prior  experiments  of  the  present 
series.  They  were  briefed  in  detail  on  the  purposes  of  the  experiment 
(except  for  such  knowledge  of  experimental  conditions  as  would  vitiate 
the  results  if  foreknown)  and  on  their  duties  (cf.  also  Appendix  I). 
Numerous  practice  problems  were  run  with  each  commander  prior  to 
the  collection  of  the  data  described  below.  These  served  the  useful 
purpose  of  accustoming  the  commanders  to  the  specific  situation  and 
of  "shaking  down"  and  solidifying  experimental  procedures. 

As  explained  in  Appendix  II  where  considerably  more  detail 
may  be  found  concerning  the  construction  of  the  experimental  threat 
evaluation  -  action  selection  problems  and  the  factors  under  investiga¬ 
tion  in  the  present  study,  it  was  considered  important  to  control  the 
effects  of  "practice"  insofar  as  their  systematic  effects  on  perform¬ 
ance  were  concerned.  While  the  main  design  was  intended  to  be  fully 
counterbalanced  with  respect  to  load  and  threat  performance  level  it 
was  not  always  possible  to  adhere  to  this  plan  for  logistic  reasons. 

The  experimenters  were  forced  by  circumstances  to  make  use  of  the 
various  commanders  when  their  time  could  be  made  available  rather 
than  vice  versa.  Added  to  the  factor  of  personnel  availability  the 
vagaries  of  equipment  malfunction  literally  forced  a  certain  amount  of 
catch -as -catch -can  scheduling.  Experience  with  and  foreknowledge  of 
the  liklihood  of  such  difficulties  in  addition  to  the  reasons  given  in  the 
Introduction,  were  further,  though  not  major,  reasons  why  no  attempt 
was  made  to  fabricate  or  execute  one  of  the  more  complex  classical 
experimental  designs. 
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RESULTS  AND  DISCUSSION 


* 


During  the  actual  data  collection  runs  detailed  records  were 
kept  of  all  the  actions  selected  by  the  commanders,  the  evaluations  of 
these  actions  and  their  outcomes.  These  records  included  assignment 
slips,  referee  sheets  (cf.  Appendix  III)  and  photographs  of  the  weapon 
status  boards.  The  three  general  headings  under  which  performance 
measures  or  indicators  fell  were  the  damage  sustained,  the  cost  in 
terms  of  defensive  weapon  usage  and  the  success  of  the  commander 
in  reducing  the  forces  of  the  enemy.  These  have  been  referred  to  in 
previous  reports  as  damage,  cost  and  kills.  With  the  records  genera¬ 
ted  during  runs  it  was  possible,  afterwards,  to  assemble  a  complete 
set  of  data  on  everything  that  happened  with  respect  to  every  track. 
Data  extracted  for  each  track  included: 

1.  Damage  data: 

(a)  Actual  problem  time  at  which  damage  was 

inflicted, 

(b)  Actual  number  of  weapons  removed  from  inventory 
as  a  result  of  damage, 

(c)  The  amount  of  time  the  damaged  site  was  out  of 

operation. 

2.  Cost  -  For  each  assignment  there  was  recorded: 

(a)  The  kind  of  defensive  weapon  that  was  assigned 
(including  site,  the  kind  of  a/c  or  missile  and  its  armament), 

(b)  The  number  of  weapons  assigned  and  the  manner  of 
employment  (i.  e.  ,  in  single  flight,  at  trail,  salvo,  etc.). 
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(c)  The  number  of  weapons  actually  used  (generally 
less  than  assigned  since  the  first  of  a  succession  of  weapons  might 


kill  and  the  remainder  be  returned  to  inventory), 

(d)  The  time  delay  in  assignment  (i.  e,  ,  actual  time 
between  onset  of  track  and  assignment  time), 

(e)  The  "time  interval"  (i,  e.  ,  which  five  minute  time 
period  during  the  problem)  in  which  the  assignment  was  made.  This 
category  of  data  was  primarily  used  for  sorting  purposes, 

(f)  The  time  to  intercept.  This  was  the  time,  in 
minutes,  between  assignment  and  outcome  whether  failure  or  success, 

(g)  Kill  probability.  This  was  the  two  digit  designation 
of  the  probability  distribution  which  was  sampled  in  assigning  a 
successful  or  unsuccessful  outcome  to  the  particular  action  selected 
(cf.  also  Appendix  HI). 

3.  "Kills"  -  For  each  track  there  were  entered: 

(a)  Overall  outcome,  i.  e.  ,  killed  or  missed  by  any  or 
all  assignments  against  it, 

(b)  Ancillary  outcome.  There  were  six  additional 
categories  of  outcome  in  additions  to  kill  and  miss.  These  included 
the  one -class  and  two -class  overmatch  (cf.  Appendix  I)  the  undermatch 
and  assignments  made  too  late  to  have  any  possibility  of  success 
(either  due  to  the  permanent  loss  of  the  target  by  the  system  or  an 
impossible  "tail  chase")  as  well  as  "previously  killed". 

In  addition  to  these  performance  data  categories  there  were 
numerous  items  of  identifying  and  categorizing  information  relating  to 
each  track.  All  of  these  (including  track  number,  threat  kind,  onset 
time,  damage  potential,  etc.)  were  entered  into  punched  cards  together 
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with  the  performance  data.  This  process  permitted  rapid  and  flexible 
data  processing.  The  results  to  be  described  below,  it  must  be  pointed 
out,  are  almost  exclusively  group  results.  That  is  they  are  summa¬ 
ries  and  averages  for  all  commanders  under  the  conditions  specified. 

It  will  be  remembered  that  the  principal  factors  whose 
effects  were  under  investigation  in  this  study  were  load,  data  quality 
and  threat  performance  levels.  Figure  3  is  a  summary  of  the  overall 
effects  of  the  load  variable.  These  results  have  been  described  and 
discussed  in  some  detail  in  a  preliminary  summary  report  of  the  pre¬ 
sent  experiment  (1).  The  general  pattern  of  load  effects  on  the  summ¬ 
ary  performance  measures  (damage,  cost  and  kills)  is  that  which 
would  be  expected.  Weapon  usage  is  seen  to  rise  with  rising  threat 
load,  but  at  a  decreasing  rate.  The  total  number  of  threats  success¬ 
fully  countered  shows  a  similar  relationship  with  load  while  damage, 
conversely,  shows  a  definite  positive  acceleration.  All  of  these 
general  findings  indicate  an  approaching  saturation  of  the  ability  of  the 
single  commander  to  keep  pace  with  the  total  increasing  threat  situa¬ 
tion.  None,  however,  shows  evidence  of  a  sharp  break  or  drastic 
deterioration.  More  detailed  treatment  of  the  three  major  classes  of 
performance  measures  as  well  as  some  additional  indices  will  be 
found  below. 

Weapon  Assignment  and  Usage: 

Table  1  presents  a  summary  of  the  percentages  of  the  total 
inventory  of  weapons  available  to  the  commanders  which  were  actually 
assigned,  on  the  average,  against  the  incoming  threats  as  a  function 
of  first  and  second  trial  at  each  load  and  separately  for  the  two  levels 
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Figure  3.  Summary  Performance:  Cost,  Kills  and  Damage 
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of  threat  performance  ("Tape  I"  and  "Tape  II").  Only  one  combination 
of  96  tracks  was  possible  with  each  tape  and  these  maximum  loads 
were  not  replicated.  The  major  difference  between  the  "Tapes"  was 
in  the  average  speed  of  the  threat  tracks  (cf.  also  Appendix  II).  It 
may  be  seen  in  Table  I  that  there  is  a  tendency  to  assign  more  weap¬ 
ons  against  essentially  the  same  number  of  threats  on  second  replica¬ 
tion  of  load  84  in  Tape  I  and  at  all  replicated  loads  in  Tape  II.  In 

TABLE  1 


Percentage  of  Weapon  Inventory  Assigned 

Load 

60  72  84  96 


Tape 

I 

II 

I 

II 

I 

II 

I  II 

Replication  1 

76 

65 

79 

78 

80 

91 

88  91 

Replication  2 

76 

74 

79 

81 

85 

95 

addition  except  for  load  60,  more  weapons  were  assigned  against  a 
given  load  (number  of  threats)  in  Tape  II  than  Tape  I. 

There  are  several  possible  factors  in  explanation  of  these 
results.  One  major  possibility  lies  in  the  order  of  presentation  of 
the  problems.  While  the  order  of  presentation  of  loads  within  a  tape 
was  counterbalanced  (working  from  low  to  high  on  first  replication 
and  from  high  back  to  low  on  second  replication)  or  scrambled  (quasi¬ 
random  order)  to  control  learning  effects,  it  is  patent  that  the  first 
replication  always  preceded  the  second.  Also,  most  frequently  the 
commanders  were  presented  with  the  Tape  I  problems  before  the 
Tape  II  problems  of  equivalent  load.  The  experience  factor  may 
account  for  some  of  the  increase  in  weapon  assignment  with  replication. 
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As  far  as  the  increase  from  Tape  I  to  Tape  II,  however,  analysis  of  the 
data  on  the  "core"  tracks  (the  set  of  identical  tracks  which  appeared 
in  both  tapes  -  cf.  Appendix  II)  did  not  reveal  any  systematic  differen¬ 
ces  in  the  numbers  of  weapons  assigned  against  them  as  a  function  of 
the  tape  in  which  they  appeared.  Other  data  described  below  also 
suggest  that  the  difference  here  between  tapes  is  not  merely  a  "prac¬ 
tice  effect. " 

Other  factors  which  tend  to  illuminate  the  finding  that  larger 
numbers  of  weapons  were  assigned  against  threats  in  Tape  II  may  be 
found  in  the  general  manner  in  which  assignments  were  made.  First, 
it  should  be  pointed  out  as  shown  in  Figure  4  that  the  two  tapes  appear 
to  produce  two  fairly  distinct  patterns  of  assignment  behavior  at  all 
but  the  lowest  load.  Note  that  a  substantially  greater  proportion  of 
the  total  number  of  threats  in  Tape  II  is  ignored  than  in  Tape  I  (i.  e.  , 
no  action  was  assigned  against  them).  One  would  expect  this  to  result 
in  the  assignment  of  fewer  weapons  in  total.  Note  also  that  less  of  the 
tracks  in  Tape  II  were  first  dealt  with  in  "raid"  assignments,  despite 
the  fact  that  nearly  equal  proportions  of  tracks  in  both  tapes  appeared 
in  "raids".  In  raid  assignments  groups  of  tracks  could  be  handled  by 
a  single  action  selection  and  the  use  of  fewer  weapons  in  total  could 
produce  the  same  number  of  kills  as  a  larger  number  of  weapons 
assigned  track -by -track  (cf.  also  Appendix  I).  The  implication  here 
is  that,  since  a  larger  proportion  of  the  threats  in  Tape  II  were  handled 
as  individual  tracks  and  since  this  mode  of  assignment  would  be  ex¬ 
pected  to  require  assignment  of  more  weapons  that  more  weapons  in 
total  would  be  assigned  against  Tape  II  threats.  The  latter  expectation 
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PER  CENT  OF  FOE  TRACKS 


Figure  4.  Proportions  of  Foe  Tracks  Npt  Acted  Upon 
and  First  Acted  Upon  by  Raid  Assignment. 
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was  borne  out  by  two  additional  replications  (beyond  those  shown  in 
Table  1)  of  60  track  problems  using  each  of  the  two  tapes.  These 
replications  were  subject  to  the  restriction  that  "raid"  type  assign¬ 
ments  were  not  permitted  and  all  assignments  had  to  be  made  on  a 
track -by -track  basis.  For  the  Tape  I,  load  60  problems  this  restric¬ 
tion  (plus  possibly  some  of  the  experience  effects  noted  above)  pro¬ 
duced  an  average  of  7%  more  weapons  assigned. 

For  the  Tape  II,  load  60  problems  the  "no  raid  assignment" 
rule  produced  an  average  of  20%  more  weapons  assigned  in  total. 

Since  in  every  instance  the  commanders  faced  both  of  the  Tape  II,  load 
60,  replications  three  and  four  before  dealing  with  the  corresponding 
Tape  I  replications  the  apparent  decrease  in  weapons  assigned  as  a 
function  of  experience  here  tends  to  vitiate  the  "learning"  hypothesis 
as  a  general  explanation  of  the  weapon  assignment  picture. 

Now  the  two  factors  cited  above  (i. e.  ,  decrease  in  weapons 
assigned  due  to  increase  in  number  of  tracks  ignored  and  that  of  in¬ 
creased  assignment  due  to  less  economical  methods  of  handling 
tracks)  are  plainly  antithetical.  Indeed,  they  might  be  expected 
further  to  have  mutually  cancelling  effects.  One  clear-cut  possibility 
of  an  explanation  of  the  generally  higher  weapon  assignment  (and  actual 
use  in  "battle")  shown  in  Figure  5  to  be  characteristic  of  the  Tape  II 
results  may  be  found  in  the  fact  that,  over  all  loads  combined,  40% 
more  weapons  were  assigned  in  fourth  and  subsequent  assignments 
(as  opposed  to  the  total  in  1st,  2nd  and  3rd  assignments)  against  Tape 
II  tracks. 

Slightly  more  weapons  also  were  assigned  in  the  first  three 
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PERCENTAGE  OF  TOTAL  INVENTORY 
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Figure  5.  Percentage  of  Counter  Weapon  Inventory  Assigned 
and  Used  as  Functions  of  Threat  Performance  and 
Load. 
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assignments,  on  the  average,  in  Tape  II  than  in  Tape  I.  This  reflects 
the  less  economical  track -by-track  type  of  assignment  which  has  been 
found  to  be  more  frequent  in  Tape  II. 

Aside  from  possible  learning  effects  due  to  order  of  presen¬ 
tation  (indications  of  which  here  are  somewhat  contradictory  and  by  no 
means  certain)  a  tentative  conclusion  may  be  reached  as  to  the  cause 
of  this  difference  in  assignment  pattern.  The  conclusion  is  that  the 
main  difference  between  the  tapes,  i.  e.  ,  threat  performance  level  - 
could  be  responsible.  While  the  threat  situation  as  defined  by  number 
of  tracks  entered  as  of  a  certain  time  was  not  greater  in  Tape  II  the 
situation  developed  somewhat  faster  in  terms  of  the  speed  with  which 
threat  vehicles  moved.  One -eighth  of  the  aircraft  tracks  in  Tape  II 
were  at  speeds  as  much  as  25%  greater  than  their  counterparts  in 
Tape  I  and  all  threats  on  the  average  moved  at  a  12%  higher  speed  in 
Tape  II  (cf.  also  Appendix  II).  The  more  rapidly  developing  situation 
(in  terms  of  track  speed)  and  the  tendency  to  make  larger  initial 
assignments  and  reassignments  which  this  could  have  produced  is  a 
probable  explanation  of  the  generally  higher  weapon  assignment  and 
usage  against  Tape  II  tracks. 

Figure  6  shows  the  rate  of  build-up  of  the  surveillance  track 
load  in  terms  of  the  average  number  of  tracks  entered  per  minute  for 
the  first  ten  minutes  of  the  problems.  These  averages  include  both 
situation  tapes  and  all  replications.  Figure  7  shows  the  problem  time 
at  which  the  commanders,  on  the  average,  had  made  a  number  of 
action  assignments  equal  to  the  total  number  of  tracks  entered  up  to 
that  time.  The  latter  is  an  indication  of  how  long  it  took  commanders. 
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Figure  6.  Rate  of  Load  Build-up  for  First 
Ten  Minutes  of  Problem. 


Figure  7.  Average  Time  Taken  by  Commanders  •> 

to  Make  a  Number  of  Assignments 
Equal  to  the  Number  of  Tracks  Present. 
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to  "catch  up"  with  the  developing  situation.  It  is  worth  noting  that, 
while  the  increase  in  load  build-up  rate  was  linear,  the  time  taken  to 
"catch  up"  with  it  exhibits  a  marked  increase  between  load  84  and  load 
96.  The  load  build-up  rate  at  which  the  abilities  of  the  commanders 
to  "keep  pace"  or  to  "catch  up"  with,  and  overtake  the  developing 
situation  appears  to  be  of  the  order  of  five  tracks  per  minute.  In  fact, 
the  actual  number  of  assignments  made  per  minute  in  the  first  five 
minutes  varied  between  4  (at  the  60  track  level)  and  6  (at  the  96  track 
level).  Since  the  problems  started  with  an  "unassigned  -  against" 
backlog  of  five  to  ten  tracks  as  a  function  of  load  this  first  time  sample 
represents  the  period  of  most  concentrated  activity  to  simply  "catch 
up".  Five  or  six  judgments  and  action  selections  of  the  sort  required 
here  is  probably  the  maximum  that  can  be  expected  per  minute. 

The  actual  usage  of  weapons  by  weapon  class  as  a  function  of 
load  is  shown  in  Figure  8.  The  numbers  used  are  quite  proportionate 
to  the  numbers  available.  At  every  load  the  largest  class  (high-yield 
aircraft)  was  most  frequently  used,  the  next  largest  class  (low-yield 
missiles)  next  most  frequently  used  and  so  forth.  In  addition,  all 
classes  are  used  in  increasing  numbers  with  higher  loads.  There  is 
some  slight  evidence  for  a  "tapering  off"  in  the  use  of  all  classes  from 
load  84  to  load  96,  which  (as  may  also  be  seen  in  Figure  3)  most 
probably  indicates  that  the  load  is  requiring  the  use  of  a  total  number 
of  weapons  approaching  the  total  number  available.  Note  that  the 
totals  in  Figure  8  would  be  uniformly  lower  than  those  in  Figure  3 
since  Figure  8  includes  weapons  used  only  as  a  result  of  the  first  three 
or  less  assignments  against  each  track  while  Figure  3  includes  all 
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Figure  8.  Average  Number  of  Weapons  Used,  by  Class, 
in  First  Three  Assignments. 
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assignments.  The  difference  is  not  great,  however,  fourth  and  sub¬ 
sequent  assignments  contributing  an  average  of  19  more  weapons  used 
at  load  60  and  an  average  of  16  more  at  load  96.  That  one  limiting 
factor  at  load  96  was  "available  inventory"  is  indicated  by  the  fact  that 
the  grand  total  of  200  weapons  (see  Figure  3)  used,  plus  36  weapons 
lost  due  to  damage  at  this  load  equals  236  whereas  the  total  available 
was  254. 

It  should  be  pointed  out  also  that  high-yield  aircraft  showed 
a  tendency  to  be  the  "weapon  of  choice"  against  raids.  In  general, 
raid-type  assignments  were  permitted  only  with  the  use  of  high-yield 
armed  weapons  -  either  missiles  or  aircraft.  The  commanders  tended, 
in  this  experiment,  to  use  high-yield  armed  aircraft  (of  which  more 
than  twice  as  many  were  available)  rather  than  missiles  against  raids. 
The  net  effect  appears  to  have  been  to  assign  and  use  the  high-yield 
aircraft  in  such  a  way  that  the  general  proportion  between  availability 
and  usage  as  noted  above,  was  maintained.  That  is,  while  the  60  to  75 
per  cent  usage  ratio  holds  for  all  classes  of  weapons  except  high-yield 
aircraft  in  track -by -track  assignments  when  raid-type  actions  are 
added  in  it  also  holds  for  high-yield  aircraft. 

In  addition  to  threat  performance  level  (cf.  "tape"  above)  and 
track  load  this  experiment  included  another  factor,  internal  to  all 
"problems"  or  threat-evaluation  action -selection  situations  faced  by 
the  commanders.  This  factor  was  data  quality.  Care  was  taken  to 
provide  an  approximate  balance  or  average  quality  of  information  pre¬ 
sented  so  that  no  particular  combination  of  other  conditions  (load,  tape 
or  replication)  would  be  affected  by  extremely  good  or  poor  overall  data 
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quality  (cf.  also  Appendix  II).  The  major  purpose  of  the  planned 
degradation  of  the  information  on  which  the  commanders'  decisions 
had  to  be  based  was  to  take  into  account  more  "realistic"  conditions. 
Previous  experiments  (2,  4)  had  presented  position  situation  and  iden¬ 
tifying  or  descriptive  data  which  were  complete  and  correct.  It  is 
hardly  to  be  expected  that  such  a  condition  would  obtain  in  actual 
field  operations. 

A  subsidiary  purpose  of  the  data  quality  variation  was  to 
obtain,  from  the  results,  some  knowledge  of  the  effects  of  incomplete 
or  incorrect  information  on  the  decision  making  behavior  of  the 
commanders.  In  general,  these  effects  were  somewhat  erratic. 

Table  2  summarizes  the  obtained  relationships  between  weapon  usage 
per  track  and  the  various  kinds  and  amounts  of  information  quality. 
Number  of  weapons  used  is  the  measure  appropriate  to  this  particular 
section.  However,  it  reflects  directly  the  number  assigned  (and 
is  most  frequently  somewhat  smaller)  which,  in  turn,  reflects  the 
attention  given  by  the  commanders  to  tracks  of  varying  data  quality. 
Generally  speaking,  there  were  no  differences  in  such  things  as 
initial  reaction  time  or  other  performance  measures  since  all  tracks 
were  presented  with  correct  and  complete  data  for  the  first  two 
minutes  of  their  lives.  Obviously  there  is  no  sense  in  studying  per¬ 
formance  against  threats  not  yet  presented. 
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TABLE  2 


/ 


Mean  Number  of  Weapons  Used  Per  Track  as 
Function  of  Data  Kind  and  Quality 


Per  Cent  Data  Complete  and  Correct 


Kind  of  Data 

10 

20 

30 

40 

50 

70 

80 

NTD1 

3.  1 

2.0 

1.8 

1.6 

2.4 

2.2 

1.8 

F2 

.9 

2.3 

1.4 

2.  1 

2.  1 

1.7 

2.6 

ANC3 

1.3 

1.9 

1.8 

1.5 

2.6 

1.8 

2.6 

F-NTD4 

1.6 

.8 

2.8 

1.6 

2.6 

2.7 

1.7 

All  2. 2 

Notes:  1.  NTD  tracks  did  not  move  for  varying  proportions 

of  their  lives. 

2.  F  tracks  faded  or  failed  to  appear  for  all  but  the 

stated  percentage  of  time. 

3.  ANC  tracks  were  supplied  with  erroneous  IFF, 

speed  or  altitude  data. 

4.  F -NTD  tracks  either  faded  or  stopped  moving, 
or  both. 

First,  it  should  be  pointed  out  that  there  were  only  very  small 
samples  of  each  category  and  amount  of  data  unreliability  within  any 
one  combination  of  other  conditions  (except  for  the  100%  complete  and 
correct  tracks  which  were  the  largest  category).  More  than  half  the 
tracks  were  presented  with  70  or  more  per  cent  of  their  data  complete 
and  correct.  For  this  reason  data  from  all  loads,  both  tapes  and  all 
replications  are  included  in  the  average  numbers  of  weapons  used 
shown  in  Table  2.  Aside  from  the  rather  great  variability  within  kinds 
of  data  or  quality  levels,  only  a  few  observations  are  possible.  First, 
above  about  50%  complete  and  correct  the  number  of  weapons  used  per 
track  seems  to  cluster  fairly  reliably  around  the  average  number  for 
100%  tracks  for  all  classes  of  information.  Also,  the  "fade"  tracks, 
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overall,  had  the  fewest  weapons  used  against  them.  It  is  safe  to  say 
that,  if  a  track  is  out  of  sight  it  will  tend  also  to  be  ignored  to  a 
greater  or  lesser  extent.  Vehicles  which  stop  in  their  tracks  (NTD 
tracks)  tend  to  elicit  the  use  of  about  as  many  weapons,  on  the  aver¬ 
age,  as  tracks  that  act  more  normally.  In  general,  however,  with 
respect  to  weapon  usage  no  really  solid  and  consistent  behavioral 
effects  emerged  except  that  even  the  very  poor  quality  tracks  attracted 
substantial  attention  and  action  from  the  commanders, 

A  sample  of  tracks  in  three  data  quality  categories  (10%,  30% 
and  100%)  was  studied  in  detail  for  the  kinds  of  weapons  which  were 
assigned  against  them.  The  general  tendency  of  the  commanders 
were  to  assign  aircraft  in  their  first  action  against  all  categories 
(60-70%  of  first  actions  used  aircraft  weapons).  Against  the  100% 
complete  and  correct  tracks  second  and  third  actions  used  missile 
counter-weapons  approximately  60%  of  the  time.  Second  and  third 
actions  against  10%  tracks,  however,  predominantly  employed  air¬ 
craft.  Against  30%  tracks  the  commanders  went  to  missiles  on  sec¬ 
ond  assignments  and  then  back  to  aircraft  on  third  assignments,  but 
in  both  cases  the  ratio  of  missile  to  aircraft  assignments  was  much 
closer  to  50-50  than  was  the  case  with  either  100%  or  10%  tracks. 

Since  missiles,  on  the  average,  had  a  higher  kill  probability  (other 
factors  such  as  delay  in  assignment,  and  number  assigned  being 
equal)  than  aircraft  in  this  study  the  tendency  to  assign  and  reassign 
aircraft  weapons  against  targets  with  very  poor  data  quality  could 
indicate  an  unwillingness  to  commit  the  scarcer  and  more  potent 
missiles  against  dubious  tracks.  When  the  commanders  were  debriefed. 
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however,  they  generally  reported  that  the  information  presented  to 
them  was  of  a  much  higher  quality  than  it  actually  was.  The  comman¬ 
ders  were  so  immersed  in  the  detailed  effort  to  deal  with  threats  one 
by  one  or  in  raid  groups  on  the  basis  of  criteria  which  were  more 
demanding  of  attention  (i.  e.  ,  threat  category,  speed,  position  relative 
to  defended  areas,  etc.)  that  they  were  unable  to  appreciate  the  more 
elusive  details  of  data  quality.  Without  special  and  attention-getting 
indication  of  data  quality  (not  present  in  this  study)  it  is  unlikely  that 
poor  quality  tracks  would  elicit  specialized  handling. 

Damage  Sustained: 

Table  3  contains  a  summary  of  all  damage  data  classified  by 
"tape"  and  load.  The  percentages  of  weapons  lost  are  based  on  the 
actual  total  numbers  removed  from  available  inventory  as  a  result  of 
hostile  penetrations  and  "bomb -drops”  as  related  to  the  total  inventory 
initially  available.  The  "down-time”  percentages  are  based  on  the 

TABLE  3 

Per  Cent  of  Weapon  Inventory  and  Per  Cent  of 
Potential  "Down-Time”  Assessed  as  Damage 


Load 


% 

Tape 

60 

72 

84 

96 

Weapons 

Lost 

I 

8.7 

7.2 

11.8 

14.  1 

II 

6.6 

7.3 

10.2 

14.  6 

% 

Down 

Time 

I 

34.  0 

29.9 

26.9 

47.  3 

II 

21.4 

25.  1 

29.7 

33.  8 
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actual  number  of  minutes  (total)  that  sites  were  closed  (i.  e.  ,  their 
weapons  were  unavailable  for  use)  divided  by  the  total  potential  closed 
time  that  was  built  into  the  problems  (cf.  also  Appendix  II).  IRBM 
and  bomber -type  threats  were  capable  of  inflicting  heavy  damage  in 
both  categories.  Fighter -bombers  could  not  cause  total  destruction 
of  a  penetrated  site  nor  long  periods  of  "down-time".  Fighter-type 
threats  were  capable  of  only  very  limited  weapon  destruction  and 
could  cause  no  "down-time".  If  no  counter -actions  whatever  had  been 
taken  a  complete  loss  of  all  weapons  in  the  initial  inventory  would 
have  ensued  and  one  or  more  sites  would  have  been  placed  "out -of 
action"  for  totals  of  120  to  200  minutes  (depending  on  load). 

While  weapon  loss  did  not  differ  consistently  or  greatly  from 
tape  to  tape  it  rose  by  a  factor  of  just  under  two  from  the  lowest  to 
the  highest  track  loads.  Since  the  potential  loss  was  100  per  cent  at 
all  loads,  the  actual  damage  sustained  (i.  e.  ,  maximum  14.  6%)  was  not 
great.  This  was  true  despite  the  fact  that  the  actions  of  the  comman¬ 
ders  did  not  differentially  kill  tracks  programmed  to  do  actual  damage 
(as  opposed  to  those  which  were  not  so  programmed)  as  will  be  seen 
below. 

"Down-time",  on  the  other  hand,  except  at  load  84,  was  great¬ 
er  for  Tape  I  problems  than  Tape  II  problems.  While  some  of  the  same 
possibilities  of  "practice  effect"  cited  in  explanation  of  the  weapon 
usage  results  above  would  also  apply  here  there  is  at  least  one  addi¬ 
tional  cause.  In  both  tapes  the  single  category  of  threat  which  caused 
the  largest  number  of  weapons  lost  in  nearly  all  instances  was  that  of 
fighter  penetrations.  These  threats  were  capable  of  causing  the  loss 
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of  only  one  weapon.  Since  the  commanders  concentrated  upon  counter¬ 
ing  the  IRBM  and  bomber-type  threats  and  tended  to  defer  or  omit 
action  against  fighter-type scomparatively  many  fighters  were  per¬ 
mitted  to  penetrate  and  cause  damage.  This  last  was  true  in  the  case 
of  both  tapes  and  all  loads.  When  other  threat  classes  were  consid¬ 
ered,  however,  closer  study  revealed  that  bomber  penetrations 
occurred  more  frequently  than  fighter  bomber  penetrations  in  the 
Tape  I  problems  whereas  the  reverse  was  true  in  the  Tape  II  results 
(see  Action  Outcome  s  below).  Thus,  while  the  programmed  penetra¬ 
tions  of  these  types  of  threats  was  equal  in  the  two  tapes,  the  actions 
of  the  commanders  were  such  that  bombers  were  prevented  from 
causing  damage  to  a  greater  extent  in  Tape  II  problems.  The  greater 
"down-time"  found  for  Tape  I  is  at  least  partly  explainable  by  this 
fact,  since  the  damage  potential  of  bombers  and  fighter -bombers 
differed  as  far  as  the  two  categories  of  damage  was  concerned. 
Fighter-bombers,  on  the  average,  were  capable  of  causing  only  about 
one -half  the  amount  of  "down-time"  as  bombers.  While  the  weapon 
destruction  potential  of  fighter -bombers  was  also  less,  many"poten- 
tials"  in  this  regard  were  of  a  percentage  type.  Thus,  in  the  event 
that  a  bomber  or  fighter -bomber  penetrated  a  defended  area  a  per¬ 
centage  of  the  weapons  then  on  the  ground  at  the  penetrated  site  would 
be  removed  from  available  inventory  for  the  remainder  of  the  run. 
Since  few  penetrations  of  either  class  (bombers  or  fighter -bombers) 
were  scheduled  before  problem  time  10-12  minutes  and  since  forty  to 
fifty  assignments  (mostly  of  more  than  one  weapon  each)  had  generally 
been  made  by  these  times  (cf.  Figure  7  above)  a  large  proportion 
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(33-50%)  of  available  weapons  had  been  committed  and/or  expended. 
Thus,  both  bombers  and  fighter -bombers  might  find  numbers  of  wea¬ 
pons  available  for  destruction  sufficiently  small  that  overall  totals 
destroyed  by  either  would  not  differ  greatly.  For  instance,  the  diff¬ 
erence  between  25%  and  50%  of  four  weapons  is  only  one  weapon.  Such 
a  difference  could  be  obscured  by  the  large  numbers  of  single -weapon 
losses  caused  by  fighter  penetrations  (cf.  above).  Therefore  the 
larger  quantities  of  "down-time"  in  Tape  I  problems  were  probably 
due  to  the  more  frequent  bomber  penetrations.  The  apparent  change 
in  tactics  by  the  commanders  (more  concentration  on  bomber  threats) 
which  is  involved  here  could  be  considered  a  species  of  "learning 
effect"  but  is  probably  related,  at  least  in  part,  to  the  pattern  of  diff¬ 
erences  between  tapes  in  weapon  assignment  and  usage  discussed 
above. 

Action  Outcomes: 

As  noted  previously  there  are  two  basic  outcomes  possible  for 
every  assignment,  either  kill  or  miss.  These  outcomes  were  selected 
from  samples  of  probability  distributions  pre -prrog rammed  on  the 
"referee  sheets"  (cf.  Appendix  III).  Certain  other  evaluations  of 
assignments  were  also  made.  If  an  aircraft  weapon  which  was  too 
slow  or  had  too  low  a  ceiling  was  assigned  it  was  an  "undermatch"  and 
would  always  miss  (cf.  also  Appendix  I).  It  was  also  possible  for  the 
commanders  to  make  one -class  and  two -class  overmatches,  i.  e.  , 
assigning  fighters  of  a  class  appropriate  to  threats  one  or  two  catego¬ 
ries  higher  than  the  threat  in  question  in  speed  and/or  altitude.  There 
was  also  the  possibility  that  a  previous  assignment  by  the  commander 
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had  already  effected  a  kill  or  would  effect  one  prior  to  the  time  that  the 
assignment  in  question  could  do  so.  Lastly,  there  were  two  categories 
of  untimely  assignment  --  the  ’’tail -chase"  wherein  the  target  had 
turned  back  and  was  running  away  before  the  weapon  could  be  scrambled 
and  the  "too  late"  wherein  the  target  was  already  so  far  away  that  it 
could  not  be  intercepted  in  the  time  available  or  within  the  surveillance 
area. 

The  ove rail  kill  rates  did  not  differ  from  tape  to  tape  except  in 
one  instance.  Table  4  gives  a  summary  of  the  percentages  of  threats 
killed  classified  by  load  and  tape.  Except  at  load  60  there  are  only 
very  small  differences  between  tapes  in  the  kill  rate.  The  definite 
difference  at  load  60  in  favor  of  the  Tape  II  results  is  a  function  of  a 
slightly  higher  average  number  of  kills  per  run  (1.  7  more  per  run)  and 

TABLE  4 

Percentage  of  Threats  Killed 

Load 


60 

72 

84 

96 

Tape  I 

79.  5 

82.9 

77.  7 

70.  2 

Tape  II 

86.7 

83.  6 

77.  5 

70.4 

a  slightly  lower  average  number  of  threats  per  run  (2.  8  less).  This 
was  an  accident  of  the  construction  of  the  tapes  and  the  selection  of 
blocks  of  tracks  used  to  make  up  the  load  60  problems.  Although  there 
were  only  six  non -ho  stile  tracks  in  either  tape,  those  in  Tape  II  appear¬ 
ed  more  frequently  in  the  load  60  problems.  Note  the  general  trend  to 
decreasing  percentage  of  kills  as  load  is  increased.  The  number  of 
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kills  (cf.  Figure  3  above)  actually  rises  with  load  but  at  a  decreasing 
rate,  as  witnessed  by  the  sharp  drop  in  percentage  kills  between  load 
84  and  load  96. 

When  the  threat  tracks  were  divided  into  those  programmed  to 
penetrate  (and  inflict  damage)  and  those  not  so  programmed,  there  is 
no  large  or  systematic  difference  between  the  two  classes.  Table  5 
presents  a  summary  of  the  percentages  of  "real"  (potential  damage) 
and  "apparent"  (no  potential  damage)  threats  which  were  acted  against 

TABLE  5 

Percentages  of  "Real"  vs.  "Apparent"  Threats 
Assigned  Against  and  Killed* 

Load 


60 

72 

84 

96 

Tape 

I  Real 

83 

88 

86 

81 

Apparent 

91 

91 

88 

76 

II  Real 

92 

92 

91 

90 

Apparent  93  93 

*(#  killed^#  assigned  against)  X  100 

89 

85 

and  killed.  It  must  be  remembered  that  not  all  threats  (of  either 
class)  were  acted  against  in  either  tape  so  that  the  kill  percentages 
here  are  all  higher  than  those  found  in  Table  4  for  the  same  loads. 
Except  for  load  60  in  Tape  I  and  load  96  in  both  tapes  the  differences 

i 

between  real  and  apparent  threat  kill  rates  are  quite  small.  In  Tape 
I  a  higher  percentage  of  apparent  threats  was  killed  except  at  the  high¬ 
est  load  while  in  Tape  II  the  same  is  true  at  the  lowest  loads  and  the 
reverse  is  true  at  the  higher  loads.  The  generally  lower  kill  figures 
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for  Tape  I  as  compared  to  Tape  II  are  accounted  for  by  two  factors  - 
the  relative  numbers  of  threats  acted  against  and  the  numbers  of 
weapons  assigned.  In  general,  in  the  Tape  I  problems,  a  somewhat 
smaller  number  of  assignments  and  re -assignments  was  spread  over 
a  somewhat  greater  number  of  threats  acted  against.  This  was  the 
choice  of  the  commanders,  since  the  numbers  and  kinds  of  threats 
were  equivalent  in  the  two  tapes  (with  the  minor  exception  in  Tape  II 
load  60  already  noted). 

The  really  important  point  here  is  that  "real"  and  "appar¬ 
ent"  threats  were  acted  against  almost  indiscriminately.  As  was 
noted  above,  commanders  tended  to  ignore  fighter-type  threats  or  to 
withhold  action  against  them  until  the  more  potent  vehicles  had  been 
acted  upon.  They  did  not,  as  has  just  been  shown,  either  learn  to 
recognize  (through  having  seen  some  of  the  tracks  before  in  other 
loads)  or  single  out  for  special  attention  (due  to  the  apparent  heading 
of  the  threat  or  other  data  available  to  them)  those  tracks  which  would 
actually  cause  damage  if  not  killed.  The  general  tendency  of  the 
commanders  to  concentrate  -  in  number,  size  and  potency  of  assign¬ 
ments  -  on  bombers  first,  then  on  fighter -bombers  and  lastly  on 
fighter -type  threats  is  indicated  further  by  the  general  kill -to -miss 
ratios  shown  in  Table  6.  If  the  first  three  assignments  against 
tracks  are  considered  (these  would  include  over  90%  of  all  assign¬ 
ments)  it  may  be  seen  that  nearly  two  out  of  three  assignments  against 
bombers  were  successful.  Table  6  presents  the  ratios  of  numbers 
of  assignments  resulting  in  kills  to  those  resulting  in  misses  for  the 
three  general  classes  of  aircraft  threats  by  load  and  tape.  The 
variability  induced  by  load  is  neither  great  nor  consistent.  It  should 
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be  noted,  however,  that  the  ratios  for  bomber  type  threats  are  the 
highest  (i.  e. ,  more  of  the  assignments  against  bombers  were  success¬ 
ful)  and  those  for  fighters  are  the  lowest.  The  small  but  apparently 

TABLE  6 

Ratio  of  Assignments  Resulting  in  Kills  to  Those 
Resulting  in  Misseti  Intercepts 

Load 


60 

72 

84 

96 

All 

Tape  I 

1.6 

1.2 

1.8 

1.4 

1.52! 

Bombers 

Tape  II 

1.9 

1.9 

1.  5 

1.7 

1.76 

Tape  I 

1.0 

.9 

1.  0 

.9 

00 

O' 

• 

Fighter 

Bombers 

Tape  II 

.9 

LI 

.9 

.7 

.94 

Tape  I 

.8 

.8 

.8 

1.  1 

I*- 

00 

• 

Fighters 

Tape  II 

.9 

1.0 

.  8 

.7 

.90 

*  Slight  difference  between  average  over  all  loads  and  average  of 
individual  averages  are  due  to  rounding  errors. 

reliable  difference  in  favor  of  the  Tape  II  assignments  against  bom¬ 
bers  helps  to  illuminate  the  explanation  previously  advanced  for  the 
superior  damage  prevention  ("down-time")  in  the  Tape  II  problems. 

In  general,  a  higher  kill-to-miss  ratio  would  result  from  the  use  of 
more  weapons  in  the  assignments  or  higher  capability  weapons  or 
both. 

Before  leaving  the  subject  of  kills  and  misses  it  should  also 
be  noted  that  the  ground-to-ground  and  air-to-ground  IRBM  threats 
were  very  well  countered  by  the  commanders  (and  caused  compara¬ 
tively  little  damage).  However,  these  threats  also  were  acted  against 
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without  much  discrimination  in  the  "real  vs.  apparent"  threat  sense 
noted  above  for  aircraft.  The  assignments  against  them  succeeded  in 
ratios  of  nearly  four  to  one.  As  will  be  seen  below,  this  type  of  threat 
was  acted  against  very  quickly.  In  addition,  the  anti-missile  counter 
weapons  were  extremely  lethal.  These  two  factors  tended  to  result  in 
high  proportions  of  kills. 

In  summary  of  the  overall  outcome  data  it  may  be  said  that 
the  threat  evaluative  activities  of  the  commanders,  as  inferred  from 
these  results,  were  of  a  rudimentary  sort.  In  the  absence  of  other 
information  on  which  to  base  relative  or  absolute  evaluations  of  threat 
the  commanders  acted  primarily  upon  the  identified  object  class. 

While  some  (comparatively  poor)  information  as  to  heading  was  to  be 
had  from  persistence  trails  the  main  items  of  situation  data  available 
were  position,  speed  and  altitude.  Unless  some  p re -evaluation  or 
weighted  combination  of  these  and/or  other  relevant  factors  were  to  be 
supplied  to  the  commanders,  it  could  be  predicted  that  their  actions 
would  be  based  upon  broad  criteria  weighed  only  crudely  and  selected 
under  the  pressures  of  load  and  time. 

When  other  categories  of  outcome  are  considered  some 
interesting  features  emerge.  Table  7  contains  a  summary  of  the  aver¬ 
age  numbers  of  the  six  different  categories  defined  above  which  were 
found  per  run  as  a  function  of  load.  Since  differences  between  tapes 
here  were  neither  large  nor  consistent.  Table  7  contains  averages 
through  both  tapes.  The  single -class  overmatch  was  more  frequent 
than  the  two -class  overmatch  at  all  loads.  The  unusually  high  numbers 
of  overmatches  at  load  84  were  most  likely  due  to  temporary 


TABLE  7 


Average  Number  per  Problem  Run  of  Outcomes 
Other  than  Kill  or  Miss 


Load 

Outcome 

60 

72 

84 

96 

Single -class  overmatch 

1.  1 

1*3 

3.0 

1.4 

Two -class  overmatch 

.8 

.9 

1.6 

1.2 

Previously  killed 

12.  1 

6.4 

5.  5 

4.0 

Unde  rmatch 

1.2 

1.6 

1.  8 

1.9 

Tail  -chase 

.4 

.4 

.4 

.6 

Too-late 

1.5 

1.0 

2.  2 

2.4 

unavailabilities  of  lower  performance  weapons  (e.  g. ,  Red  and  Blue 
class  aircraft  -  cf.  Appendix  I)  caused  by  base  closure  as  a  result  of 
hostile  penetration.  The  small  total  numbers  of  overmatches  and 
undermatches  overall,  however,  are  definite  indications  that  the 
commanders  were  taking  care  in  the  application  of  matching  criteria 
--  selecting  weapons  which  had  the  appropriate  speed  and  altitude 
capabilities  and  neither  "wasting”  high  capability  weapons  on  lower 
capability  threats  nor  failing  to  kill  by  assigning  weapons  without  the 
performance  necessary  to  intercept.  While  this  matching  behavior 
tends  to  deteriorate  as  a  function  of  increased  load  it  does  not  do  so 
seriously.  It  may  be  noted  also  that  the  higher  loads  produced  some¬ 
what  more  untimely  (Tail-chase  or  Too-late)  assignments.  More 
attention  will  be  given  to  this  question  below  under  "Delay  Times." 

\The  largest  single  category  of  outcome  to  be  seen  in  Table  7 
is  that  of  "previously  killed."  These  were  "unnecessary"  assignments. 
Most  frequently  they  consisted  of  the  assignment  of  ground  -to -air 
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missile  counter  weapons  against  targets  to  which  previous  assign¬ 
ments  of  aircraft  had  been  made.  If  the  outcome  of  the  earlier  assign¬ 
ment  had  not  yet  become  known  to  the  commander  and  if  the  target 
was  still  present  a  back-up  or  additional  assignment  was  made  fairly 
frequently.  Such  occurrences  were  more  frequent  at  the  lower  loads, 
where  the  inventory  appeared  more  ample  and  where  the  pressure  of 
load  permitted  additional  assignments.  Even  here,  however,  where 
the  commanders  were  indeed  somewhat  ’’wasteful"  of  their  weapons 
such  assignments  only  constituted  a  fairly  small  proportion  of  all 
assignments  except  at  the  lowest  load.  It  may  be  inferred  that,  when 
a  substantial  superfluity  of  weapons  appears  to  exist,  the  human 
decision  maker  tends  to  let  considerations  of  damage  prevention  and 
destruction  of  enemy  forces  override  considerations  of  economy  of 
weapons. 

Action  Kill  Probabilities: 

The  kill  probability  of  any  action  selected  is,  in  a  certain 
sense,  an  overall  evaluation  of  its  quality.  True,  it  reflects  the  qual¬ 
ity  of  other  factors  besides  the  choice  itself  in  "real  life",  including 
the  capability  of  the  weapon  selected,  the  quality  of  the  intercept 
direction,  etc.  In  this  experiment,  however,  these  other  factors  were 
not  allowed  to  affect  the  probability  of  kill.  It  was  (as  may  be  seen  in 
considerable  detail  in  Appendices  I  and  III)  strictly  determined  by  the 
kind,  the  number  and  the  armament  of  weapons  chosen  by  the  comm¬ 
ander  and  the  time  at  which  the  action  was  selected.  In  this  study, 
therefore,  the  kill  probabilities  of  selected  actions  reflect  the  quality 
of  the  choice  made  by  the  commanders.  Table  8  contains  a  summary 
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TABLE  8 


Percentage  of  Actions  and  Average  Kill 
Probability  as  Functions  of  Assignment 
Mode 

Load 


60 

72 

84 

96 

Action  Mode 

% 

Pr 

%* 

% 

Pr 

%* 

Pr 

Single  Flight 

66 

.  51 

64  . 

50 

6  5 

.49 

63  . 

50 

Two  -at-T  rail 

22 

.  57 

24  . 

59 

23 

.  58 

23  . 

58 

Three  -at-T  rail 

12 

.  63 

13  . 

66 

12 

.66 

13  . 

64 

^Failure  to  total  100%  is  due  to  rounding  errors. 

of  the  proportions  of  the  three  major  classes  of  assignment  made  and 
the  average  kill  probabilities  which  they  involved.  Both  tapes  are 
averaged  here  since  there  were  no  consistent  or  large  differences 
between  tapes.  The  major  point  of  interest  in  Table  8  is  the  remark¬ 
able  consistency  in  both  mode  of  assignment,  and  average  action 
quality  (as  indicated  by  the  kill  probabilities)  from  load  to  load.  About 
two -thirds  of  all  actions  selected  were  in  the  "single  flight"  mode. 
These  were  most  frequently  of  the  "single  weapon  against  single  threat" 
type,  although  some  consisted  of  two  or  even  three  weapons.  The 
latter  was  more  frequently  true  in  the  case  of  missile  counter  weapons. 
The  reason  for  this  type  of  assignment  being  the  most  frequent  is  that 
it  was  the  only  permissable  mode  for  missile  assignment  and  it  was 
the  most  economical  mode  for  aircraft  assignment  (cf.  also  Appendices 
I  and  III).  That  is,  if  two  or  three  aircraft  were  to  be  used  in  a  single 
assignment  it  was  advantageous  to  assign  them  "at  trail"  since  an 
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improved  probability  of  kill  was  insured  thereby  (cf.  Appendix  III  on 
nReferee  Sheets").  Note,  however,  that  the  average  kill  probability 
of  these  "single  flight"  assignments  was  the  lowest,  averaging  .  50 
ove  rail. 

Assignments  of  aircraft  counter-weapons  "two  at  trail"  were 
next  most  frequent  (just  under  one-quarter  of  all  assignments)  follow¬ 
ed  by  "three  at  trail"  which  constituted  about  one -eighth  of  the  assign¬ 
ments.  There  is,  as  might  be  expected  due  to  compounding  of  prob¬ 
abilities  in  trail  assignments,  a  rise  in  average  kill  probability  to  .  58 
for  two  at  trail  actions  and  further  to  .65  for  three  at  trail  actions. 

As  noted  above,  the  important  point  here  is  the  consistency 
of  the  commanders.  The  general  tenor  of  their  assignments  was  not 
affected  by  load  either  in  kind  or  in  overall  quality.  The  major  effect 
of  load,  as  has  been  pointed  out  above,  was  on  the  number  of  assign¬ 
ments  and  the  major  effect  of  the  difference  between  tapes  was  on  the 
kinds  and  numbers  of  threats  against  which  actions  were  taken.  The 
only  evidence  for  adaptation  of  assignment  kind  to  conditions  appeared 
in  replications  3  and  4  of  load  60,  both  tapes,  wherein  the  comman¬ 
ders  made  more  "three  at  trail"  assignments  (18%)  and  less  "two  at 
trail"  (13%)  than  under  other  conditions.  This  reversal  is  most  likely 
in  some  way  due  to  the  requirement  in  these  replications  that  threats 
be  acted  against  only  track -by -track  and  not  as  raids.  The  prohibition 
of  raid-type  actions  eliminated  the  inherent  advantage  of  such  actions 
(the  ability  to  obtain  the  same  kill  probability  against  two  tracks  as 
against  one  with  the  same  number  and  kind  of  weapons,  cf.  Appendix 
I)  and  evidently  induced  the  use  of  more  three  at  trail  assignments 
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in  compensation.  The  point  is  that  load  alone  did  not  affect  the  kind  of 
action  selected  by  the  commanders.  It  would  be  interesting  to  discover 
the  manner  in  which  this  pattern  might  have  changed  if  other  factors 
were  varied  (such  as  kinds  and  number  of  weapons  available)  since 
the  elimination  of  raid-type  assignments  appeared  here  to  have  had 
some  effects. 

Assignment  Delays: 

The  last  performance  measure  which  will  be  discussed  is  tha^t 
of  delay  in  assignment.  This  was  defined  as  the  time  elapsed  between 
the  initial  appearance  of  a  threat  and  the  time  an  action  was  assigned 
against  it.  Table  9  contains  a  summary  of  the  delay  time  data  includ¬ 
ing  both  "tapes"  and  all  replications.  The  difference  between  "tapes" 


TABLE  9 

Average  and  Variability  of  Delay  in 
First  Assignment  (Minutes) 


Single  Assignments: 


Load 


60 

72 

84 

96 

Mdn  Q 

Mdn 

Q 

Mdn 

Q 

Mdn 

Q 

IRBM's 

.6  .3-.  9 

.4 

.2-.  7 

.6 

.4-1.1 

.6 

Bombers 

.8  .5-2.4 

.7 

.3-2.2 

1.  0 

.  5-3.2 

.9 

.6-2.1 

Fighter 

Bombers 

.7  .4-1.2 

.8 

.  5-1.3 

.9 

.  5-2.  5 

.9 

.6-1.8 

Fighters 

.7  .4-2.0 

.7 

.4-2.  1 

.9 

.  5-1.8 

.9 

.  5-1.8 

Raid  Assignments: 

Bombers 

.6  .5-.  9 

.  5 

.  3-.  7 

.7 

.5-1.0 

.6 

.4-1.2 

Fighter 

Bombers 

• 

o 

o 

1 

H- » 

• 

►— » 

.7 

.  5-1.0 

.8 

.7-1.2 

.9 

.  6-1.  5 

and  replications  were  neither  large  nor  consistent.  The  upper  section 
of  the  table  presents  the  median  (mdn)  delay  times  and  the  ranges 
within  which  the  middle  50%  of  delays  fell  (Q)  for  tracks  assigned 
against  one -by -one.  The  lower  section  of  the  table  gives  similar  data 
for  tracks  assigned  against  in  multiple  or  "raid"  assignments. 

The  two  most  noticeable  features  of  the  delay  times  are  that, 
for  tracks  acted  against  singly,  the  quickest  "reactions"  were  found 
for  IRBM  threats  and  that  reactions  to  bombers  and  fighter -bombers 
in  raids  were  generally  faster  than  when  handled  singly.  It  is  evident 
that  the  commanders  were  taking  account  of  two  factors  here.  In  the 
case  of  the  missile -type  threats  time  was  recognized  to  be  of  the 
essence.  Rapid  action  was  important  because  of  the  speeds  of  these 
targets  -  initial  assignments  had  to  be  made  quickly  and  they  were. 

The  variability  in  reaction  time  was  smaller  for  this  class  of  target 
than  for  any  other  when  handled  one  at  a  time.  In  the  case  of  tracks 
in  raids  the  raids  apparently  were  perceived  as  more  threatening  than 
single  tracks  and  in  addition  offered  the  commanders  the  opportunity 
to  act  against  several  tracks  simultaneously  and  thus  reduce  their 
task  burden. 

Except  for  the  IRBM's,  there  are  no  large  or  consistent 
differences  in  time  delay  amongst  the  categories  of  threats  when  acted 
upon  singly.  Average  delays  are  all  small  and,  in  general,  shorter 
than  those  found  in  the  previous  experiment  (4)  where  the  same  comm¬ 
anders  were  employed.  Much  of  this  improvement  is  probably  due  to 
the  raid  assignment  method  which  helped  reduce  the  backlog  of  tracks 
to  be  acted  upon  in  rapid  fashion.  It  should  be  pointed  out  that  there  is 
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a  trend  toward  increasing  reaction  time  with  increased  load  as  might 
be  expected.  Variability  was  not  greatly  affected  by  load  in  the  single 
assignments  but  appeared  to  increase  .slightly  at  higher  loads  for  the 
raid  assignment.  The  tendency  to  concentrate  on  bombers  prior  to 
other  categories  of  aircraft  threats  which  has  been  remarked  previous¬ 
ly  is  evident  in  the  shorter  delays  (in  most  cases)  found  for  bombers 
as  opposed  to  fighter -bombers  when  handled  in  raids. 

The  conclusion  to  be  drawn  from  these  data  (as  from  some 
of  the  other  measures)  is  that  the  commanders  tended  to  react  to  and 
act  upon  broad  classes  of  threats  in  a  fairly  consistent  and  character¬ 
istic  manner.  They  were  too  pressed  by  the  overall  task  to  try  to 
make  fine,  individual  evaluations  or  discriminations.  The  actions 
they  selected  and  the  general  principles  of  action  that  they  adopted 
were  appropriate  and  effective. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


There  was  some  evidence  in  the  results  of  this  study  that  the 
general  level  of  performance  of  tracks  in  the  surveillance  situation, 
particularly  their  speeds,  influenced  the  actions  of  the  commanders 
in  a  general  way.  Against  Tape  II  threats  (higher  average  speed)  the 
commanders  exhibited  a  different  sort  of  selectivity  amongst  threats. 
While  fighter-type  threats  were  often  ignored  completely  or  action 
against  them  was  deferred  in  both  tapes,  there  was  a  tendency  for 
less  actions  against  fighters  in  Tape  II  situations.  In  addition,  the 
commanders  tended  to  concentrate  somewhat  more  on  bomber -type 
threats  in  the  higher  performance  situations.  This  general  shift  in 
tactics,  while  partly  explainable  in  terms  of  experience  effects  can¬ 
not  be  completely  accounted  for  thereby.  While  fine  discriminations 
between  threats  apparently  were  not  being  made,  the  commanders 
showed  a  highly  developed  ability  to  make  correct  and  appropriate 
"matches"  to  targets  in  their  selection  of  counter  weapons.  It  is 
likely  that  if  better  or  more  explicit  indications  of  heading  had  exist¬ 
ed  in  the  position  situation  display,  the  commanders  would  have  made 
more  detailed  discriminations  and  somewhat  fewer  general  ones  (e.  g. , 
as  between  "this  bomber  vs  that  bomber"  rather  than  "bomber  vs 
fighter -bomber"). 

Another  finding  (though  not  startingly  new)  was  that  pressure 
of  a  backlog  of  tasks  "pushed"  the  commander  toward  what  appears 
to  be  a  "natural"  limit  in  their  capacity  to  make  the  kind  of  decisions 
required  in  this  study.  This  limit  seems  to  be  fairly  stable  at  a 
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level  of  five  to  six  per  minute.  Even  where  they  were  pressing  this 
limit,  however,  the  overall  quality  of  the  actions  selected  (as  indica¬ 
ted  by  inclusive  measures  such  as  kill  probability)  did  not  deteriorate 
seriously.  Thus,  while  increasing  task  load  tended  to  increase  delays 
in  the  selection  of  counter  actions,  the  increase  in  "reaction  time" 
was  not  serious  enough  to  degrade  the  effectiveness  of  the  average 
action. 

The  effects  upon  decision  making  performance  in  this  study 
of  the  quality  of  surveillance  data  were  neither  very  great  nor  sharply 
defined.  However,  the  lowest  quality  tracks  did  appear  to  be  handled 
somewhat  differently  than  those  whose  data  were  more  correct  and 
complete.  Very  low  quality  tracks  attracted  the  use  of  less  weapons 
generally.  Undoubtedly  some  of  this  effect  is  due  to  a  fairly  obvious 
factor  -  if  you  don't  see  it  as  often  or  as  long,  you  will  not  attend  to 
it.  In  addition  to  this  effect,  however,  there  was  evidence  that  low 
quality  tracks  attracted  a  different  kind  of  reassignment.  Low  quality 
tracks  were  reattacked  with  the  use  of  aircraft  counter  weapons  much 
more  frequently  than  were  the  better  tracks  (where  missiles  were 
used  most  frequently).  While  there  may  have  been  other  logic  behind 
such  a  tactic  it  is  likely  that  commanders  were  reluctant  to  use  their 
non -re callable  weapons  against  doubtful  tracks.  In  any  case,  tracks 
whose  data  were  fifty  per  cent  or  more  complete  and  Correct  were 
handled  in  essentially  the  same  manner  as  those  whose  data  were 
perfect.  The  commanders  had  neither  the  time  nor  the  specific  infor¬ 
mation  necessary  to  thus  differentiate  on  any  explicit  basis  -  it  was 
done  through  some  ingrained,  implicit  process. 
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Finally,  it  was  found  that  economic  considerations  tend  to  be 
overlooked.  Despite  the  fact  that  the  matching  of  weapons  to  threats 
was  very  good  not  much  care  was  taken  to  conserve  weapons.  There 
was  a  definite  tendency  on  the  part  of  the  commanders  to  "use  up" 
available  weapons  and  even  squander  them  somewhat  when  the  supply 
appeared  to  be  much  greater  than  the  "demand".  This  is  probably 
perfectly  natural  in  view  of  their  really  primary  concern  -  to  avoid 
damage  and  to  decimate  the  forces  of  the  enemy.  In  practice,  however, 
it  is  probably  not  good  strategy. 

From  these  results,  several  recommendations  may  be  evolved. 

1.  Man-machine  surveillance  systems  should  (as  most  now 
do)  provide  reasonable  indications  of  heading  of  targets  to  human 
evaluators. 

2.  At  least  some  gross  indications  of  the  reliability  of  sur¬ 
veillance  data  should  be  provided  to  commanders. 

3.  The  capabilities  of  commanders  in  monitoring  and  over¬ 
riding  an  automated  or  semi-automated  decision  making  system  should 
be  investigated  in  detail. 

4.  Decision  criteria  for  man-machine  decision  makers 
should  be  developed  to  include  economic  and  logistic  considerations 
even  at  the  direct  action  or  battle  level. 

5.  Large  amounts  of  intensive  experience  in  reasonably 
realistic  conditions  and  under  high  task  loads  should  be  provided  for 
commanders  at  all  levels.  Such  experience  not  only  promotes  maxi¬ 
mum  performance,  but  may  prevent  overload  from  producing  drastic 
effects. 
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6.  The  effects  of  different  kinds  and  degrees  of  "pre -diges¬ 
tion"  or  synthesis  of  surveillance  data  on  decision  making  performance 
should  be  investigated. 
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APPENDIX  I 


Instructions  to  the  Commanders: 

The  general  mission  of  the  commander  or  decision  maker  in 
this  experiment  will  be  essentially  the  same  as  in  previous  experi¬ 
ments.  This  mission  is  threefold: 

1.  To  minimize  damage  to  defended  areas, 

2.  To  destroy  a  maximum  number  of  threatening  vehicles, 

3.  To  conserve  counter  weapons  so  as  to  consume  a  mini¬ 
mum  of  defensive  forces  consistent  with  objectives  (1)  and  (2)  above. 

The  specific  tasks  by  which  controllers  may  accomplish 
these  objectives  will  be  quite  similar  to  those  employed  in  previous 
experiments.  The  purpose  of  this  briefing  brochure  is  to  outline  some 
of  the  specific  differences  in  procedures  which  will  be  required  for 
the  present  experiment. 

A.  Threat  Vehicle  Types 

In  the  present  experiment  there  will  be  four  broad  categories 
of  threatening  vehicles.  There  will  be: 

1.  IRBM  Type  Missiles.  These  may  be  of  the  land-launched 
(therefore  first  appearing  at  the  edges  of  the  coverage  area  and  at 
speeds  of  3000  knots)  or  air  launched  (and  thus  first  appearing  further 
within  the  coverage  area  and  at  speeds  of  2000  knots).  These  will  be 
distinguished  primarily  by  extremely  high  speed.  "Decoys"  may  be 
present.  This  type  of  threat  may  be  countered  only  by  an  inventory 
of  anti-missile  missiles  of  long  range,  high  speed  and  high  capability. 


-49- 


2.  Air-Breathing  Bomber  Threats.  There  will  be  four 
broad  performance  categories  within  this  class: 

(a)  Ultra -high  Bombers  -  This  group  includes  bombers 
of  very  high  speed  (1500  knots)  as  well  as  altitude  (90,  000  feet).  These 
may  be  countered  only  with  ground-to-air  missiles  and  with  White -class 
fighte  rs. 


(b)  High  Bombers  -  This  group  includes  bombers  of 
lower  altitude  (75,  000  feet),  but  which  may  vary  in  speed  between  1000 

and  1500  knots.  The  "matching"  class  fighter  is  the  Green-type. 

■V'V  .  ■'  •  ‘  <•.'  f:  t:  Kg? 

Missiles,  ‘of  course,  may  alsojbe  used.  In  the  event  that  such  a  threat 

* 

is  "over -matched"  (by  countering  with  White -class  fighters)  the  inter¬ 
cept  will  be  timed  at  Green-class  speeds  and  altitudes,  but  taking  into 
consideration  the  higher  kill  capabilities  of  the  White -class  weapons. 

(c)  Medium  Bombers  -  This  group  of  vehicles  includes 
bombers  whose  altitude  is  lower  still  (60,  000  feet)  but  whose  speed 
range  overlaps  with  that  of  the  High  bomber  types  (800-1200  knots). 

The  "matching"  class  of  interceptors  is  the  Blue-type.  As  noted  above, 
any  overmatch  will  be  run  at  Blue -class  speeds,  but  will  be  given  the 
advantage  of  the  higher  class  kill  probabilities.  Again,  all  types  of 
ground-to-air  missiles  may  also  be  used  against  this  type  of  threat. 

(d)  Low  Bombers  -  These  vehicles  have  an  altitude  cap¬ 
ability  of  50,  000  feet  and  speeds  of  the  order  of  800  knots.  The 
"matching"  class  is  the  Red-type.  Overmatches  will  be  handled  as 
mentioned  above.  Ground-to-air  missiles  may  be  used. 
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3.  Fighter  Bombers.  There  are  three  classes  of  fighter- 


bombe  rs: 

(a)  High  -  Altitude  55,000  feet,  speed  600-800  knots. 

The  matching  class  fighters  are  the  Red  IR  and  Blue -types. 

(b)  Me dium  -  Altitude  30-45,  000  feet,  speed  5-600 
knots.  Red-type  fighters  are  the  matching  class. 

(c)  Low  -  Altitude  50, 000  feet,  speed  600  knots. 
Red-type  fighters  are  the  matching  class.  (Note:  There  was  also  one 
very  low  altitude  fighter -bomber  raid  at  5,  000  feet  about  which  the 
commanders  were  not  forewarned).  Ground-to-air  missiles  may  be 
used  against  all  types  of  fighter-bombers.  Overmatches  will  be 
handled  as  noted  above  under  "Bombers". 

4.  Fighters.  Threatening  fighters  will  be  of  four  classes: 

(a)  Ultra  -High  -  90,  000  feet,  12-1500  knots 
White  fighters  are  the  matching  class. 

(b)  High  -  75,  000  feet,  10-1200  knots 
Green  fighters  are  the  matching  class. 

(c)  Me  dium  -  60,  000  feet,  8-1200  knots 
Blue  fighters  are  the  matching  class. 

(d)  Low  -  30-50,  000  feet,  600  knots 
Red  fighters  are  the  matching  class. 

The  general  characteristics  of  the  threat  vehicles  which  have  been 
listed  above,  while  generally  definitive,  will  not  be  absolutely  iron¬ 
clad.  There  may  be  some  threats  at  very  low  altitude  and  comparative¬ 
ly  high  speed.  In  addition,  the  auxiliary  data  associated  with  a  track 
will  be  subject  to  error  (see  section  on  "Intelligence",  below). 
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B.  Counter  Weapon  Types  and  Sites 

The  counter  weapons  available  to  the  controller  will  be  of 
three  broad  types.  They  will  be  located  at  ten  sites,  two  such  sites 
in  each  of  five  larger  defended  areas.  The  forward  or  enemy-ward 
areas  will  be  designated  areas  1  and  2.  The  intermediate  areas  some 
distance  directly  behind  the  latter  will  be  designated  areas  3  and  4. 

The  single  rear  area  will  be  designated  area  5.  All  counter  weapons 
(except  Red  class  aircraft)  may  have  either  high  yield  or  low  yield 
armament.  The  long  range  missiles  with  high  yield  warheads  will  have 
a  ten  mile  blast  radius.  High  yield  aircraft  and  high  yield  short  range 
missiles  will  have  five  mile  blast  radius.  Low  yield  weapons  are  of 
the  contact  type.  The  general  weapon  classes  will  be: 

1.  Anti-Missile  Missiles.  These  weapons  are  of  high  speed 
(3000  knots)  and  high  accuracy  and  are  armed  with  high  yield  warheads 
giving  them  a  very  high  kill  probability.  They  will  be  located  in  area 
5,  the  rearward  area.  They  will  be  housed  in  hardened  fashion,  some 
ready  to  fire  and  some  in  standby  and  will  be  relatively  impervious  to 
all  except  high  yield  direct  hits.  They  will  be  found  in  silo -type 
shelters.  "Standby"  missiles  require  three  minutes  to  be  converted 

to  "ready"  status.  While  this  will  be  taken  care  of  automatically  it 
should  be  remembered  that  rapid  consumption  of  ready  missiles  could 
result  in  delay  in  firing  due  to  transition  from  "standby"  to  "ready". 

2.  Other  Ground-to-Air  Missile s. 

(a)  Long  Range  Missiles  -  These  weapons  are  of  two 
types  -  high  yield  warhead  and  low  yield  warhead  armed.  They  will 
be  found  in  hardened  and  semi -hardened  bases  located  in  areas  3  and 
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4.  They  are  capable  of  kill  anywhere  within  the  coverage  area  against 
any  type  of  target  except  IRBM's.  The  high  yield  types,  of  course, 
have  higher  kill  probabilities. 

(b)  Mobile  Missiles  -  These  weapons  have  similar 
characteristics  and  capabilities  to  the  Long  Range  type  except  for 
lower  kill  probabilities  and  that  they  will  be  found  on  mobile  launchers 
in  the  forward  areas  1  and  2  and  may  be,  therefore,  somewhat  more 
vulnerable  to  attack  than  other  Long  Range  Types.  Their  "mobility" 
is  more  a  matter  of  definition  here  than  a  matter  for  concern  or 
manipulation  on  the  part  of  the  decision  maker. 

(c)  Short  Range  Missiles  -  These  weapons  have  a 
maximum  range  of  100  miles  and  will  be  located  at  sites  in  areas  3 
and  4  in  such  a  way  that  their  coverages  overlap  and  also  span  the 
width  of  the  surveillance  area.  They  will  be  of  both  high  and  low 
yield  armament  types  and  will  be  of  somewhat  lower  capability 
generally  than  the  Long  Range  missiles.  They  will  be  found  in  both 
semi-hardened  and  "soft"  configurations. 

3.  Interceptor  Aircraft. 

(a)  White  Class  Fighters  (90,000  feet)  -  These  are 
a  very  high  performance,  very  high  capability  interceptor  some  of 
which  will  carry  higher  yield  armament  than  others  (resulting  in  a 
higher  kill  probability).  As  noted  above  they  are  the  matching  class 
counter  weapon  for  ultra -high  performance  threats.  They  will  be 
found  in  area  5  only.  White  fighters  may  not  be  used  against  targets 
below  70,  000  feet. 
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(b)  Green  Class  Fighters  (75,000  feet)  -  These  are  a 
high  performance  interceptor,  some  of  which  will  have  high  yield 
armament.  They  will  be  located  at  bases  in  areas  3  and  4. 

(c)  Blue  Class  Fighters  (60,000  feet)  -  These  are 
medium  performance  weapons  and  may  be  high  or  low  yield  armed. 
They  will  be  based  in  both  areas  3  and  4. 

Note:  White,  Green,  and  Blue  class  fighters  will  be  found  in  both 
"hard"  and  "soft"  configurations.  The  hardening  will  be  of  the  "cave" 
or  "tunnel"  type  and  there  may  be  some  degree  of  delay  in  the  launch¬ 
ing  of  hardened  aircraft. 

(d)  Red  Class  Fighters  -  These  are  low  performance 
weapons.  They  may  be  armed  with  either  IR  seeker  missiles  (usable 
to  55,000  feet)  or  with  guns  (49,000  feet).  In  general,  they  are  the 
matching  class  weapon  for  low  performance  threats.  However,  there 
is  one  exception.  The  missile-armed  Red  fighters  may  use  the  "snap- 
up"  attack.  Infra-red  seeker  armed  fighters  of  this  class,  therefore, 
may  be  employed  against  threatening  vehicles  which  are  either  above 
the  altitude  capability  of  the  Red  fighters  (i.  e.  ,  up  to  55,  000  feet)  or 
above  the  speed  capability  of  Red  fighters  (max.  600  knots),  but  not 
both.  That  is,  Red  fighters  may  be  employed  against  some  high  and 
medium  class  threats  except  where  the  threat  is  above  both  50,  000 
feet  and  600  knots.  Gun -armed  Red  fighters  may  only  be  used  against 
low  performance  threats. 

C.  Scoring  Procedures 

In  general,  where  "matching"  class  fighters  or  ground-to- 
air  missiles  are  assigned  by  the  controller,  scoring  will  present  no 
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problems.  An  "undermatch"  (assignment  of  a  Blue  class  weapon  to  a 
threat  for  which  the  matching  class  is  White,  for  example)  will  be 
scored  as  a  miss.  Overmatching,  as  described  above  will  ordinarily 
increase  kill  probability  but  will  give  no  advantage  in  intercept  time 
(or  return -to  -  inventory  time). 

D.  Weapon  Assignment  Modes  and  Procedures 

One  of  the  most  important  points  to  be  remembered  concern¬ 
ing  weapon  assignments,  aside  from  selection  of  appropriate  weapons, 
will  be  the  standardization  of  the  assignment  message  format.  Stan¬ 
dardization  in  this  respect  is  of  the  utmost  importance  from  the 
experimental  and  from  the  practical  operational  point  of  view.  Exper¬ 
imentally,  only  standard  messages  can  be  handled  with  the  required 
speed  and  accuracy  by  the  clerks  and  referees.  However  convenient 
a  non-standard  message  is  to  the  controller,  it  shows  down  or  con¬ 
fuses  the  execution  of  the  assignment  resulting,  ultimately  in  penalty 
to  the  controller.  In  addition,  future  systems  will  require  communica¬ 
tion  with  machines  in  language  the  machines  can  understand.  Non¬ 
standard  messages  are  understood  and  executed  even  less  well  (i.  q.  , 
not  at  all)  by  machines  than  by  men  (as  in  the  present  experiment). 
Therefore,  an  essential  part  of  procedure  will  be  the  use  of  rigid 
message  formats.  The  assignment  message  will  consist  of  the  track 
number  assigned  against,  the  weapon  kind  to  be  used,  the  site  to  be 
drawn  from,  its  armament  category  and  the  number  to  be  used  (e.  g. , 
Track  84,  "Green  three  high  two"). 
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As  in  previous  experiments,  weapons  may  be  assigned  one, 
two  or  three  at  a  time  against  individual  threats.  In  addition,  "trail" 
type  assignments  of  two  or  three  successive  aircraft  may  be  used. 

Due  to  the  actual  vagaries  of  surveillance  data  and  close  control 
performance  (i.  e.  ,  a  prior  failure  of  control  might  still  add  data  to 
improve  a  subsequent  attempt),  trail  type  assignments  generally 
result  in  a  higher  kill  probability  overall.  Weapons  not  actually  em¬ 
ployed  will  be  returned  to  inventory  as  before. 

In  this  experiment  there  will  be  groups  of  threats  in  the  form 
of  "raids".  The  number  of  individual  tracks  in  a  raid  may  vary  from 
two  to  ten  or  more.  Raids  will  appear  strung  out  across  a  "front", 
individual  tracks  five  to  ten  miles  apart.  Tracks  in  a  raid  will  bear 
the  same  "personal  identification"  and  a  raid  size  indication  will 
appear  at  the  right-hand  end  of  the  auxiliary  data  readout  panel.  Zero 
here  indicates  "unknown"  raid  size  (10  or  mere  tracks),  one  indicates 
"small"  (2-3  tracks),  two  indicates  "medium"  (4-6  tracks)  and  three 
indicates  "large"  (7-9  tracks). 

Raids  may  be  countered  by  making  single  assignments 
between  pairs  of  tracks  in  a  raid  with  high-yield  missiles  or  high 
yield  armament  aircraft.  Such  assignments  will  result  in  the  same 
liklihood  of  killing  both  members  of  a  pair  as  if  two  separate  assign¬ 
ments  had  been  made,  one  to  each  individual  track.  In  making  such 
assignments  a  track  number  from  the  counterclockwise  end  of  the 
raid  front  and  the  raid  designator  (e.  g. ,  B-2)  should  be  given  to  the 
scramble  clerk,  followed  by  the  weapon  type,  weapon  site,  armament 
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and  total  number  of  weapons  to  be  used  against  the  raid  as  well  as  the 
manner  in  which  it  is  desired  that  weapons  be  used  (e.  g.  ,  single 
weapons  between  tracks,  two  at  trail  between  tracks,  etc,).  Raid 
traffic  may  also  be  handled  track  by  track  individually.  Long  range 
missile  fire  may  be  accomplished  on  a  1/1,  2/1,  or  3/1  salvo  basis, 
with  successive  salvos  not  permitted  until  after  intercept  outcome 
data  is  fed  back.  Short  Range  Missiles  may  be  assigned  for  success¬ 
ive  "salvo"  firing  in  numbers  not  to  exceed  3  per  salvo.  They  will 
be  fired  one  minute  apart.  Probably,  however,  all  missiles  so 
assigned  will  be  removed  from  inventory  at  time  of  assignment.  Un¬ 
fired  missiles  will  be  returned  to  inventory. 

Kill  probabilities  for  White  type  aircraft  will  vary  from  .  3 
to  .  8,  depending  on  number  assigned,  armament  type  and  delay  in 
assignment  as  before.  Green  class  fighters  will  have  kill  probabilities 
of  .  2  to  .7,  Blue  class  .1  to  .6,  and  Red  class  .  1  to  .  5.  The  more 
weapons  assigned  (up  to  3,  either  together  or  in  "trail"),  the  higher 
the  potency  of  the  armament  and  the  sooner  after  onset  of  the  threat 
the  higher  the  probability  of  kill. 

Short  range  missiles  may  be  fired  on  specific  assignment 
or  may  be  ordered  to  fire  autonomously  at  specific  track  numbers  as 
they  come  into  range  (100  miles)  of  the  site  selected.  When  such  an 
autonomous  fire  is  ordered  it  will  be  done  automatically  by  the 
experimenters. 

Areas  1  and  2  are  located  in  the  forward  region  and  are  not 
hardened.  They  are  therefore  comparatively  vulnerable  to  damage 
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Their  weapons  should  be  used  early  and  the  bases  may  be  sacrificed 
or  evacuated  in  advance  of  and  in  favor  of  protecting  rearward  areas. 

Bases  may  be  evacuated  (i.  e.  ,  aircraft  scrambled  to  air¬ 
borne  status)  not  more  than  one  at  a  time.  Lost  fighter  probabilities 
will  be  applied  to  all  evacuated  fighters.  Weapons  may  not  be  evacuated 
at  a  rate  of  more  than  three  per  minute.  Weapons  may  be  ordered 
from  a  state  of  "soft"  readiness  into  hardened  shelters  at  the  same  rate. 

E.  Intelligence 

1.  Certain  classes  of  threats  will  be  more  likely  high  yield 
carriers  than  others. 

(a)  50%  of  IRBM  threats  will  be  high  yield  carriers 

(b)  35%  of  ultra -high  and  high  level  bombers 

(c)  15%  of  medium  level  bombers 

(d)  10%  of  low  level  bombers 

(e)  25%  of  fighter  bombers  will  carry  high  yield  weapons. 

2.  Decoys  may  be  present  to  the  extent  of: 

(a)  50%  of  IRBM' s  and 

(b)  10%  of  high  level  bombers. 

3.  Countermeasures  may  be  expected.  These  will  tend  to 
jam  sensors,  communications,  weapon  direction  and  homing  systems. 
This  will  result  in  stopping,  fading  and  jumping  of  track  position  data 
and  some  scrambling  of  auxiliary  data.  When  in  doubt  as  to  the  quality 
of  track  data  or  apparent  contradiction  or  change  of  ancillary  data  the 
controller  may  inquire  and  will  be  informed  of  the  quality  of  the  data 
available  on  any  given  track  (in  broad  categories  from  "very  poor"  to 

"excellent"). 


-58- 


APPENDIX  II 


SECTION  1 

Track  Situation  Recordings: 

1.  Two  basic  tape  recordings  of  track  situations  and  auxi¬ 
liary  data  were  prepared.  Each  tape  contained  96  complete  tracks 
(except  as  noted  below)  in  "blocks"  of  12  targets.  The  data  for  each 
track  consisted  of  its  life  history  of  positions  (i.  e.  ,  its  flight  path) 
and  the  identifying  or  descriptive  auxiliary  information  connected 
with  it  (i.  e. ,  IFF,  type,  speed,  altitude,  raid  number  if  any,  and 
raid  size  if  any).  The  proportions  of  tracks  of  different  types  will  be 
found  listed  below.  Descriptions  of  their  detailed  characteristics 
may  be  found  in  Appendix  I. 

2.  Each  block  of  12  tracks  could  be  selected  for  display  or 
completely  suppressed.  This  permitted  the  number  of  tracks  presen¬ 
ted  in  a  problem  situation  to  be  "solved"  by  the  commander  to  be 
varied  from  12  to  96  in  multiples  of  twelve.  Track  numbers  were 
scrambled  in  systematic  fashion  so  that  there  was  no  discernible 
order  which  would  tend  to  relate  any  track  to  others. 

3.  There  were  36  tracks  (three  blocks)  which  were  common 
to  both  tapes.  That  is,  these  identical  tracks  in  all  respects  appear¬ 
ed  in  both  tapes. 

4.  The  remaining  tracks  were  programmed  to  have  diff¬ 
erent  general  characteristics  in  each  tape.  The  60  tracks  which  were 
specific  to  Tape  I  were  mainly  air-breathing  threats  within  a  speed 
range  of  800  to  1200  knots  and  an  altitude  range  of  50  to  75  thousand 
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feet  plus  a  small  number  of  IRBM’s  in  the  3000  knot  class.  Tape  II 
tracks  included  some  air-breathers  with  speeds  up  to  1500  knots  and 
altitudes  up  to  90,000  feet  plus  IRBM's  as  before.  The  average  speeds 
of  each  performance  class  were  higher  in  Tape  II. 

5.  The  track  paths  were  drawn  up  on  paper  to  meet  the 
pre-selected  specifications  of  the  experimental  conditions  and,  using 
target  generators  and  associated  equipment  (2,  5)  were  recorded  on 
magnetic  tape.  This  insured  almost  absolute  duplicability  of  the 
"stimulus"  material  from  time  to  time  and  from  subject  so  subject. 
Different  combinations  of  blocks  were  displayed  for  different  total 
loads  and  for  the  same  loads  when  presented  on  different  occasions. 
Experience  has  shown,  that  under  the  circumstances  of  these  experi¬ 
ments,  little  or  no  specific  learning  of  the  stimulus  situation  takes 
place. 

6.  In  both  tapes  pre-programmed  degrees  of  data  reliability 
were  applied  to  each  track.  Both  tapes  contained  equal  proportions 

of  tracks,  the  data  on  which  were  10%,  20%,  etc.,  "reliable"  (complete 
and  correct)  in  ten  per  cent  steps  to  100  per  cent.  The  unreliability 
introduced  was  of  two  kinds.  One  type  was  in  track  position  data 
wherein  tracks  disappeared  or  stood  still  for  the  appropriate  propor¬ 
tions  of  their  track  lives.  Approximately  three -fourths  of  the  un¬ 
reliability  was  of  this  type.  The  second  type  of  unreliability  was 
introduced  into  auxiliary  track  data  as  erroneous  or  missing  identifi¬ 
cation,  speed  or  altitude.  Blocks  of  tracks  were  approximately 
balanced  for  the  total  amount  of  uncertainty  so  that  the  average  data 
quality  in  any  single  problem  was  of  the  order  of  66%  complete  and 
correct,  with  a  range  of  62  to  67  per  cent. 
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7.  Constitution  of  the  Situation  Tapes  by  Threat  Kind: 

Number 


Kind 

Performance 

Tape  I 

Tapi 

IRBM's 

"Real" 

2 

4 

"Decoy" 

4 

2 

Bombers 

Ultra -High 

0 

6 

High 

12 

10 

Medium 

12 

10 

Low 

12 

10 

F  ighte  r  - 

High 

12 

6 

Bombers 

Me  dium 

6 

12 

Low 

6 

6 

Fighters 

Ultra-High 

0 

6 

High 

8 

6 

Me  dium 

8 

6 

Low 

8 

6 

F  riends 

6 

6 
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8.  Constitution  of  Situation  Tapes  by  Data  Certainty  Rate: 

Number  of  Tracks 

%  Data  Complete  &  Correct  Tape  I  Tape  II 


10 

4 

4 

20 

5 

5 

30 

5 

5 

40 

7 

7 

50 

11 

11 

60 

11 

11 

70 

12 

12 

80 

13 

13 

90 

13 

13 

100 

15 

15 

Distribution  of  these  tracks  was  approximately  balanced  over  threat 
kinds  and  through  blocks  of  tracks  so  as  to  make  it  possible  to  con¬ 
struct  complete  problems  of  various  total  loads  of  very  similar  char¬ 
acteristics  in  these  respects. 

9.  Fifty  per  cent  of  the  hostile  tracks  in  both  tapes  (nearly 
all  tracks  were,  in  fact,  hostile  or  potentially  hostile  (i.  e.  ,  unknown) 
were  programmed  to  inflict  damage  if  allowed  to  penetrate  the  defen¬ 
ded  areas.  Possible  damage  ranged  from  the  loss  of  one  defensive 
weapon  to  total  destruction  of  the  weapon  site.  In  addition,  damage 
could  effect  the  closing  of  a  site  (no  weapons  could  be  scrambled  from 
that  site)  for  periods  ranging  from  three  minutes  to  indefinitely.  If 
all  tracks  capable  of  inflicting  damage  were  allowed  to  penetrate  100% 
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damage  (i.  e.  ,  total  loss  of  all  defensive  weapons)  would  have  ensued. 
This  last  was  true  at  all  loads  (60-96  tracks)  used  for  data  purposes. 

SECTION  2 
Experimental  Design: 

The  "design"  of  the  present  experiment  was  not  of  any  com¬ 
plete  classical  sort.  Such  would  have  been  most  impractical  in  terms 
of  the  magnitude  of  effort  required  and  did  not  seem  to  be  necessary 
to  the  aims  of  the  program.  The  variables  were: 

1.  Track  Situation  Load.  For  data  purposes  there  were 
four  such:  60,  72,  84,  and  96  total  tracks  appearing  over  a  45  minute 
period.  In  all  cases  about  one -third  of  the  total  were  present  by  the 
end  of  the  first  five  minutes  and  90%  had  entered  by  problem  time 

25  minutes. 

2.  Operating  Performance  Range  of  Threats.  This  variable 
was  defined  by  the  two  tape  recorded  track  situations  described  above. 
Tape  I  included  threats  of  a  more  restricted  speed  and  altitude  per¬ 
formance  than  those  in  Tape  II.  The  object  here  was  to  ascertain  the 
effects,  of  a  faster  vs  a  slower  moving  and  developing  situation  on  the 
performance  measures  selected. 

3.  Data  Quality.  This  was  a  variable  which  was  internal  to 
all  other  combinations  of  conditions  and  approximately  equally  weight¬ 
ed  in  each  combination  so  that  it  would  not  have  undue  differential 
effects  on  any  one  or  another  combination  of  "tape"  and  "load". 

A  fourth  variable  might  be  considered  to  be  that  of  "Comman¬ 
ders"  or  "Subjects".  However,  the  data  for  all  subjects  were  com- 
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bined  in  treatment  and  this  variable  was  not  examined  separately. 

Attempts  were  made  to  control  learning  effects  by  counter¬ 
balancing  and/or  mixing  of  the  order  of  presentation  of  conditions 
within  each  subject.  Attempts  were  made  to  insure  reliability  by 
replication  of  most  combinations  of  conditions.  Each  commander 
(except  one)  faced  various  combinations  of  60  track  loads  from  each 
tape  four  times.  Each  commander  faced  two  different  combinations 
of  72  and  84  track  loads  from  each  tape.  Since  there  was  only  one 
combination  of  96  tracks  in  each  tape,  this  load  was  not  replicated. 

On  the  first  two  replications  of  60  tracks,  commanders  were 
permitted  to  employ  "raid"  assignments  (cf.  Appendix  I)  but  on  the 
last  two  replications  only  track -by -track  assignments  were  permitted. 
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APPENDIX  IE 


SECTION  1 

Weapon  Inventory  and  Status  Board: 

Figure  1  is  a  drawing  of  the  weapon  status  board  as  it  appear¬ 
ed  at  the  beginning  of  every  problem  run.  The  names  (Red,  Blue, 
Mobile,  Long,  etc.)  refer  to  the  classes  of  weapons  available  to  the 
decision  maker.  The  numbers  outside  the  blocks  (on  either  side  of 
the  weapon  kind  name)  refer  to  the  sites  at  which  such  weapons  were 
available.  The  lettering  across  the  tops  of  the  larger  subdivided 
blocks  refer  to  the  state  of  readiness.  For  aircraft  (color  names  left 
hand  side  of  board)  these  were  RH  meaning  "Ready  in  Hardened  Han¬ 
gars",  RS  for  "Ready  in  Soft  Locations",  AA  for  "Available  (already) 
Airborne",  and  SY  for  "Standby"  (to  become  available  after  some 
delay).  The  letters  H,  S,  and  SY  on  the  missile  side  of  the  board 
stand  for  "Hard",  "Soft"  and  "Standby"  with  the  same  meanings.  The 
letters  defining  rows  stand  for  the  type  of  armament  carried  by  the 
weapons,  IR  for  infrared  seeker  rockets,  G  for  guns,  H  for  high-yield 
blast  armament,  L  for  low-yield  armament.  The  numbers  in  the 
boxes  refer  to  the  quantities  of  the  specified  weapons  available  at  the 
start  of  the  problem.  These  were  adjusted  by  status  board  keepers 
as  assignments  were  executed  and  weapons  were  utilized. 

Weapons  were  scrambled  from  "Soft"  storage  until  exhausted 
then  from  hardened  storage.  Commanders  were  required  to  so  state 
specifically  when  they  desired  to  divert  airborne  available  fighters. 
Returning  fighters  were  placed  in  standby  for  standard  intervals,  then 
placed  in  a  ready  status. 
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Figure  1. 
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SECTION  2 
Referee  Sheets: 

Figures  2  and  3  are  samples  of  the  referee  outcome  evalua¬ 
tion  forms  which  were  used  in  this  experiment.  Figure  2  is  an  aircraft 
counter-weapon  referee  sheet  and  Figure  3  a  missile  referee  sheet, 
both  for  the  same  specific  track.  Entries  are  explained  as  follows  for 
Figure  3: 

1.  Upper  right  corner: 

B 1  50/800  (24-29-34-45)  04. 

B1  stands  for  low  performance  hostile  bomber. 

50/800  stands  for  an  altitude  of  50  thousand  feet,  speed 

of  800  knots. 

24-29-34-45  stands  for  the  onset  time  of  the  track  and 
the  ends  of  the  three  critical  time  periods  during  its  life,  i.  e.  ,  its 
first  and  second  five  minute  periods  and  its  time  of  disappearance 

l*: 

(unless  previously  killed). 

04  is  the  identifying  number  of  the  track. 

2.  Upper  right  corner: 

80-87-94-04-11-73  stand  for  the  numbers  of  all  tracks 
in  the  raid  of  which  this  track  was  a  part  reading  clockwise  along  the 
raid  front  as  they  appeared  in  the  situation  display. 

3.  Left  side: 

4  (12-9-7)  stands  for  site  4  and  the  times  taken  to  inter¬ 
cept  (or  fail  to  intercept)  as  a  function  of  the  time  period  during  the 
life  of  the  track  during  which  the  assignment  was  made.  As  an  exam¬ 
ple,  if  an  assignment  of  Blue  type  aircraft  was  made  against  track  04 
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Figure  2 
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Figure  3.  Sample  Missile  Referee  Sheet 
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at  time  30  (the  second  time  period,  between  29  and  34  (cf.  (1)  above), 
the  kill  or  miss  would  take  place  at  time  30  plus  9  or  time  39  and  the 
outcome  feedback  would  be  accomplished  at  that  time. 

4.  3  (5-3-TL)  stands  for  site  3,  same  class  of  aircraft  and 
the  intercept  times.  The  shorter  times  here  indicate  that  track  04  is 
at  all  times  closer  to  site  3.  The  "TL"  indicates  "too  late".  Thus, 
track  04  had  penetrated  the  defended  area  at  time  34  (the  beginning  of 
the  third  critical  time  perbd  of  the  track)  and  even  a  kill  would  fail 
to  prevent  damage. 

5.  The  upper  blocked-off  areas  opposite  these  site  and  time 
labels  were  both  for  Blue -type  aircraft,  found  at  sites  3  and  4.  The 
lower  blocks  have  similar  labels  (2  (15-13-10),  Red,  1  (7-5-8))  indica¬ 
ting  the  appropriate  time  quantities  for  Red-class  aircraft  at  sites  1 
and  2. 

6.  The  columns  of  X' s  and  blanks  are  random  samples  from 
distributions  the  probability  of  an  "X"  (as  opposed  to  "no  X")  in  which 
ranged  in  decrements  of  .  03  in  succeeding  columns  from  .99  for  the 
column  farthest  to  the  left  of  the  sheet  to  .12  at  the  extreme  right. 

The  columns  are  arranged  in  sets  of  three.  Within  the  rectangular 
blocking  these  sets  of  three  columns  are  further  grouped  in  threes. 
The  lefthand  set  of  nine  columns  within  the  blocked-off  section  was 
used  in  selecting  outcomes  for  assignments  of  high-yield  armed  wea¬ 
pons  and  the  right  hand  set  of  nine  for  low-yield  armament.  Within 
either  set  of  nine  columns,  the  left-most  three  were  used  in  selecting 
outcomes  for  assignments  of  three  weapons  against  a  threat,  the 
center  set  for  two  weapon  assignments  and  the  right  most  set  for 
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single  weapon  assignments.  Within  each  set  of  three  columns,  the 
left  most  was  used  to  evaluate  assignments  made  in  the  first  time 
period  of  the  life  of  the  track,  the  center  column  for  assignments  in 
the  second  time  period  and  the  right-hand  column  for  assignments 
made  during  the  remainder  of  the  life  of  the  track.  The  single  weapon 
set  of  three  columns  was  used  for  evaluating  "trail"  assignments. 

(see  below). 

7.  To  follow  two  specific  assignments  against  the  sample 
track  (numer  04)  as  examples,  consider: 

(a)  An  assignment  made  at  time  30  and  consisting  of 
two  Blue -class  fighters  from  Base  3  with  high-yield  armament.  The 
small  black  square  in  Figure  2  indicates  the  outcome  which  would  be 
selected.  It  was  selected  as  the  first  (from  the  top)  of  those  outcomes 
allocated  to  Base  3  (blocking  opposite  the  entry  3  (5-3-TL),  from  the 
left-hand  set  of  nine  columns  (high-yield  armament),  the  center  group 
of  three  columns  (two  weapons)  and  the  center  column  of  that  group 
(second  time  period  in  the  life  of  the  track  -  between  29  and  34  min¬ 
utes).  This  outcome  was  an  "X"  or  a  "kill"  and  it  would  take  place  at 
time  33  or  assignment  time  (30)  plus  3  minutes,  the  intercept  time 
pre-programmed  for  assignments  in  the  second  time  period  for  this 
track. 

(b)  An  assignment  made  at  time  27  of  two  Red-class 
fighters  with  infra-red  seeker  missiles  from  Base  2  to  be  scrambled 
"at  trail"  (i.e.  ,  one  followed  by  a  second  one  minute  later).  The 
small  black  circle  in  Figure  1  indicates  the  outcome.  To  select  an 
outcome  for  this  assignment  the  single -weapon  group  of  columns 
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(right  most  group),  left  most  column  (first  time  period),  Base  2 
(upper  of  the  lower  set  of  rectangles  )  Red  IR  set  of  nine  columns  (left 
hand  set)  was  entered.  The  circle  here  is  around  the  first  outcome 
in  that  column.  It  happened  to  be  an  "X"  or  kill.  The  second,  unused, 
Red  fighter  would  be  returned  to  inventory  as  "airborne  available"  in 
this  case.  If  the  first  entry  had  been  a  blank,  the  first  fighter  would 
have  missed  and  the  next  entry  below  it  in  the  same  column  would  be 
used  to  score  the  outcome  for  the  second  "trail"  fighter. 

8.  Note  that  the  longer  an  assignment  was  delayed,  the 
fewer  weapons  were  assigned  and  the  lower  capability  the  armament 
class  or  weapon  class  assigned  the  lower  (i.  e. ,  farther  to  the  right 
of  the  "table")  was  the  kill  probability.  Note  also  that  the  cumulative 
probability  of  an  "X"  in  the  single  weapon  columns  is,  mathematically 
greater  than  that  of  an  "X"  in  the  corresponding  two  weapon  and  three 
weapon  columns.  For  example,  if  the  probability  of  kill  of  a  single 
weapon  is  .  50  and  the  kill  probability  of  two  of  the  same  weapons  at 
the  same  time  is  .60  (which  is  the  way  probabilities  were  selected 
for  this  experiment,  i.  e.  ,  a  difference  in  probability  of  .  07  between 
corresponding  columns  in  adjacent  sets  of  three  columns)  the  mathe¬ 
matics  of  the  "trail"  probability  of  two  weapons  is  as  follows: 

First  weapon:  P(kill)  =  .  50  P(no  kill)  =  .  50 
Second  weapon:  P(kill)  =  .  50  P(no  kill)  =  .  50 
P(neither  kills)  =  .  5  x  .  5  =  .25 
P(at  least  one  kill)  =  1.  00  -.25  -  .75 
The  kill  probability  of  trail  type  assignments,  therefore,  was  always 
higher  than  that  of  the  same  number  of  weapons  in  a  "single  flight" 
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or  simultaneously. 

9.  The  missile  referee  sheets  had  similar  markings  (cf. 
Figure  3)  L4  and  L3  signifying  long  range  missiles  from  sites  4  and 
3,  S4  and  S3  similarly  for  short  range  missiles  and  Ml  and  M2  signi¬ 
fying  mobile  missiles  at  areas  1  and  2.  "FT"  followed  by  two  numbers 
(e.  g.  ,  FT  37-44)  signifies  the  time  period  during  which  autonomous 
fire  was  permitted.  The  other  data  at  the  top  right  are  the  same  as 
those  on  the  aircraft  sheet.  Scoring  was  done  in  the  same  manner 
except  that  "trail"  fire  was  not  permitted. 

10.  It  should  be  noted  that  the  same  action,  selected  at  the 
same  time  against  the  same  track  would  have  the  same  outcome  every- 
time  -  i.  e.  ,  no  matter  which  decision  maker  selected  the  action.  This 
was  necessary  for  two  reasons,  first  to  simplify  scoring  by  the  ref¬ 
erees  to  the  point  that  it  could  be  done  accurately  within  the  time 
available  and  secondly,  because  the  number  of  runs  required  to  collect 
the  data  if  a  completely  free  operation  of  the  probabilities  was  per¬ 
mitted  would  have  been  prohibitive. 
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five  to  six  selections  per  minute.  No  drastic 
deterioration  of  performance  was  found  at  the  high¬ 
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